Multi-channel audio encoding and decoding with multi-channel transform selection

ABSTRACT

An audio encoder and decoder use architectures and techniques that improve the efficiency of multi-channel audio coding and decoding. The described strategies include various techniques and tools, which can be used in combination or independently. For example, an audio encoder performs a pre-processing multi-channel transform on multi-channel audio data, varying the transform so as to control quality. The encoder groups multiple windows from different channels into one or more tiles and outputs tile configuration information, which allows the encoder to isolate transients that appear in a particular channel with small windows, but use large windows in other channels. Using a variety of techniques, the encoder performs flexible multi-channel transforms that effectively take advantage of inter-channel correlation. An audio decoder performs corresponding processing and decoding. In addition, the decoder performs a post-processing multi-channel transform for any of multiple different purposes.

RELATED APPLICATION INFORMATION

This application claims the benefit of U.S. Provisional PatentApplication Ser. No. 60/408,517, filed Sep. 4, 2002, the disclosure ofwhich is incorporated herein by reference.

The following U.S. provisional patent applications relate to the presentapplication: 1) U.S. Provisional Patent Application Ser. No. 60/408,432,entitled, “Unified Lossy and Lossless Audio Compression,” filed Sep. 4,2002, the disclosure of which is hereby incorporated by reference; and2) U.S. Provisional Patent Application Ser. No. 60/408,538, entitled,“Entropy Coding by Adapting Coding Between Level and Run Length/LevelModes,” filed Sep. 4, 2002, the disclosure of which is herebyincorporated by reference.

TECHNICAL FIELD

The present invention relates to processing multi-channel audioinformation in encoding and decoding.

BACKGROUND

With the introduction of compact disks, digital wireless telephonenetworks, and audio delivery over the Internet, digital audio has becomecommonplace. Engineers use a variety of techniques to process digitalaudio efficiently while still maintaining the quality of the digitalaudio. To understand these techniques, it helps to understand how audioinformation is represented and processed in a computer.

I. Representation of Audio Information in a Computer

A computer processes audio information as a series of numbersrepresenting the audio information. For example, a single number canrepresent an audio sample, which is an amplitude value (i.e., loudness)at a particular time. Several factors affect the quality of the audioinformation, including sample depth, sampling rate, and channel mode.

Sample depth (or precision) indicates the range of numbers used torepresent a sample. The more values possible for the sample, the higherthe quality because the number can capture more subtle variations inamplitude. For example, an 8-bit sample has 256 possible values, while a16-bit sample has 65,536 possible values. A 24-bit sample can capturenormal loudness variations very finely, and can also capture unusuallyhigh loudness.

The sampling rate (usually measured as the number of samples per second)also affects quality. The higher the sampling rate, the higher thequality because more frequencies of sound can be represented. Somecommon sampling rates are 8,000, 11,025, 22,050, 32,000, 44,100, 48,000,and 96,000 samples/second.

Mono and stereo are two common channel modes for audio. In mono mode,audio information is present in one channel. In stereo mode, audioinformation is present in two channels usually labeled the left andright channels. Other modes with more channels such as 5.1 channel, 7.1channel, or 9.1 channel surround sound (the “1” indicates a sub-wooferor low-frequency effects channel) are also possible. Table 1 showsseveral formats of audio with different quality levels, along withcorresponding raw bitrate costs.

TABLE 1 Bitrates for different quality audio information Sample DepthSampling Rate Raw Bitrate Quality (bits/sample) (samples/second) Mode(bits/second) Internet 8  8,000 mono   64,000 telephony Telephone 811,025 mono   88,200 CD audio 16 44,100 stereo 1,411,200

Surround sound audio typically has even higher raw bitrate. As Table 1shows, the cost of high quality audio information is high bitrate. Highquality audio information consumes large amounts of computer storage andtransmission capacity. Companies and consumers increasingly depend oncomputers, however, to create, distribute, and play back high qualitymulti-channel audio content.

II. Processing Audio Information in a Computer

Many computers and computer networks lack the resources to process rawdigital audio. Compression (also called encoding or coding) decreasesthe cost of storing and transmitting audio information by converting theinformation into a lower bitrate form. Compression can be lossless (inwhich quality does not suffer) or lossy (in which quality suffers butbitrate reduction from subsequent lossless compression is moredramatic). Decompression (also called decoding) extracts a reconstructedversion of the original information from the compressed form.

A. Standard Perceptual Audio Encoders and Decoders

Generally, the goal of audio compression is to digitally represent audiosignals to provide maximum signal quality with the least possible amountof bits. A conventional audio encoder/decoder [“codec”] system usessubband/transform coding, quantization, rate control, and variablelength coding to achieve its compression. The quantization and otherlossy compression techniques introduce potentially audible noise into anaudio signal. The audibility of the noise depends on how much noisethere is and how much of the noise the listener perceives. The firstfactor relates mainly to objective quality, while the second factordepends on human perception of sound.

FIG. 1 shows a generalized diagram of a transform-based, perceptualaudio encoder (100) according to the prior art. FIG. 2 shows ageneralized diagram of a corresponding audio decoder (200) according tothe prior art. Though the codec system shown in FIGS. 1 and 2 isgeneralized, it has characteristics found in several real world codecsystems, including versions of Microsoft Corporation's Windows MediaAudio [“WMA”] encoder and decoder. Other codec systems are provided orspecified by the Motion Picture Experts Group, Audio Layer 3 [“MP3”]standard, the Motion Picture Experts Group 2, Advanced Audio Coding[“AAC”] standard, and Dolby AC3. For additional information about thecodec systems, see the respective standards or technical publications.

1. Perceptual Audio Encoder

Overall, the encoder (100) receives a time series of input audio samples(105), compresses the audio samples (105), and multiplexes informationproduced by the various modules of the encoder (100) to output abitstream (195). The encoder (100) includes a frequency transformer(110), a multi-channel transformer (120), a perception modeler (130), aweighter (140), a quantizer (150), an entropy encoder (160), acontroller (170), and a bitstream multiplexer [“MUX”] (180).

The frequency transformer (110) receives the audio samples (105) andconverts them into data in the frequency domain. For example, thefrequency transformer (110) splits the audio samples (105) into blocks,which can have variable size to allow variable temporal resolution.Small blocks allow for greater preservation of time detail at short butactive transition segments in the input audio samples (105), butsacrifice some frequency resolution. In contrast, large blocks havebetter frequency resolution and worse time resolution, and usually allowfor greater compression efficiency at longer and less active segments.Blocks can overlap to reduce perceptible discontinuities between blocksthat could otherwise be introduced by later quantization. Formulti-channel audio, the frequency transformer (110) uses the samepattern of windows for each channel in a particular frame. The frequencytransformer (110) outputs blocks of frequency coefficient data to themulti-channel transformer (120) and outputs side information such asblock sizes to the MUX (180).

For multi-channel audio data, the multiple channels of frequencycoefficient data produced by the frequency transformer (110) oftencorrelate. To exploit this correlation, the multi-channel transformer(120) can convert the multiple original, independently coded channelsinto jointly coded channels. For example, if the input is stereo mode,the multi-channel transformer (120) can convert the left and rightchannels into sum and difference channels:

$\begin{matrix}{{{X_{Sum}\lbrack k\rbrack} = \frac{{X_{Left}\lbrack k\rbrack} + {X_{Right}\lbrack k\rbrack}}{2}},} & (1) \\{{X_{Diff}\lbrack k\rbrack} = {\frac{{X_{Left}\lbrack k\rbrack} - {X_{Right}\lbrack k\rbrack}}{2}.}} & (2)\end{matrix}$Or, the multi-channel transformer (120) can pass the left and rightchannels through as independently coded channels. The decision to useindependently or jointly coded channels is predetermined or madeadaptively during encoding. For example, the encoder (100) determineswhether to code stereo channels jointly or independently with an openloop selection decision that considers the (a) energy separation betweencoding channels with and without the multi-channel transform and (b) thedisparity in excitation patterns between the left and right inputchannels. Such a decision can be made on a window-by-window basis oronly once per frame to simplify the decision. The multi-channeltransformer (120) produces side information to the MUX (180) indicatingthe channel mode used.

The encoder (100) can apply multi-channel rematrixing to a block ofaudio data after a multi-channel transform. For low bitrate,multi-channel audio data in jointly coded channels, the encoder (100)selectively suppresses information in certain channels (e.g., thedifference channel) to improve the quality of the remaining channel(s)(e.g., the sum channel). For example, the encoder (100) scales thedifference channel by a scaling factor ρ:{tilde over (X)} _(Diff) [k]=ρ·X _(Diff) [k]  (3),where the value of ρ is based on: (a) current average levels of aperceptual audio quality measure such as Noise to Excitation Ratio[“NER”], (b) current fullness of a virtual buffer, (c) bitrate andsampling rate settings of the encoder (100), and (d) the channelseparation in the left and right input channels.

The perception modeler (130) processes audio data according to a modelof the human auditory system to improve the perceived quality of thereconstructed audio signal for a given bitrate. For example, an auditorymodel typically considers the range of human hearing and critical bands.The human nervous system integrates sub-ranges of frequencies. For thisreason, an auditory model may organize and process audio information bycritical bands. Different auditory models use a different number ofcritical bands (e.g., 25, 32, 55, or 109) and/or different cut-offfrequencies for the critical bands. Bark bands are a well-known exampleof critical bands. Aside from range and critical bands, interactionsbetween audio signals can dramatically affect perception. An audiosignal that is clearly audible if presented alone can be completelyinaudible in the presence of another audio signal, called the masker orthe masking signal. The human ear is relatively insensitive todistortion or other loss in fidelity (i.e., noise) in the masked signal,so the masked signal can include more distortion without degradingperceived audio quality. In addition, an auditory model can consider avariety of other factors relating to physical or neural aspects of humanperception of sound.

The perception modeler (130) outputs information that the weighter (140)uses to shape noise in the audio data to reduce the audibility of thenoise. For example, using any of various techniques, the weighter (140)generates weighting factors (sometimes called scaling factors) forquantization matrices (sometimes called masks) based upon the receivedinformation. The weighting factors in a quantization matrix include aweight for each of multiple quantization bands in the audio data, wherethe quantization bands are frequency ranges of frequency coefficients.The number of quantization bands can be the same as or less than thenumber of critical bands. Thus, the weighting factors indicateproportions at which noise is spread across the quantization bands, withthe goal of minimizing the audibility of the noise by putting more noisein bands where it is less audible, and vice versa. The weighting factorscan vary in amplitudes and number of quantization bands from block toblock. The weighter (140) then applies the weighting factors to the datareceived from the multi-channel transformer (120).

In one implementation, the weighter (140) generates a set of weightingfactors for each window of each channel of multi-channel audio, orshares a single set of weighting factors for parallel windows of jointlycoded channels. The weighter (140) outputs weighted blocks ofcoefficient data to the quantizer (150) and outputs side informationsuch as the sets of weighting factors to the MUX (180).

A set of weighting factors can be compressed for more efficientrepresentation using direct compression. In the direct compressiontechnique, the encoder (100) uniformly quantizes each element of aquantization matrix. The encoder then differentially codes the quantizedelements relative to preceding elements in the matrix, and Huffman codesthe differentially coded elements. In some cases (e.g., when all of thecoefficients of particular quantization bands have been quantized ortruncated to a value of 0), the decoder (200) does not require weightingfactors for all quantization bands. In such cases, the encoder (100)gives values to one or more unneeded weighting factors that areidentical to the value of the next needed weighting factor in a series,which makes differential coding of elements of the quantization matrixmore efficient.

Or, for low bitrate applications, the encoder (100) can parametricallycompress a quantization matrix to represent the quantization matrix as aset of parameters, for example, using Linear Predictive Coding [“LPC”]of pseudo-autocorrelation parameters computed from the quantizationmatrix.

The quantizer (150) quantizes the output of the weighter (140),producing quantized coefficient data to the entropy encoder (160) andside information including quantization step size to the MUX (180).Quantization maps ranges of input values to single values, introducingirreversible loss of information, but also allowing the encoder (100) toregulate the quality and bitrate of the output bitstream (195) inconjunction with the controller (170). In FIG. 1, the quantizer (150) isan adaptive, uniform, scalar quantizer. The quantizer (150) applies thesame quantization step size to each frequency coefficient, but thequantization step size itself can change from one iteration of aquantization loop to the next to affect the bitrate of the entropyencoder (160) output. Other kinds of quantization are non-uniform,vector quantization, and/or non-adaptive quantization.

The entropy encoder (160) losslessly compresses quantized coefficientdata received from the quantizer (150). The entropy encoder (160) cancompute the number of bits spent encoding audio information and passthis information to the rate/quality controller (170).

The controller (170) works with the quantizer (150) to regulate thebitrate and/or quality of the output of the encoder (100). Thecontroller (170) receives information from other modules of the encoder(100) and processes the received information to determine a desiredquantization step size given current conditions. The controller (170)outputs the quantization step size to the quantizer (150) with the goalof satisfying bitrate and quality constraints.

The encoder (100) can apply noise substitution and/or band truncation toa block of audio data. At low and mid-bitrates, the audio encoder (100)can use noise substitution to convey information in certain bands. Inband truncation, if the measured quality for a block indicates poorquality, the encoder (100) can completely eliminate the coefficients incertain (usually higher frequency) bands to improve the overall qualityin the remaining bands.

The MUX (180) multiplexes the side information received from the othermodules of the audio encoder (100) along with the entropy encoded datareceived from the entropy encoder (160). The MUX (180) outputs theinformation in a format that an audio decoder recognizes. The MUX (180)includes a virtual buffer that stores the bitstream (195) to be outputby the encoder (100) in order to smooth over short-term fluctuations inbitrate due to complexity changes in the audio.

2. Perceptual Audio Decoder

Overall, the decoder (200) receives a bitstream (205) of compressedaudio information including entropy encoded data as well as sideinformation, from which the decoder (200) reconstructs audio samples(295). The audio decoder (200) includes a bitstream demultiplexer[“DEMUX”] (210), an entropy decoder (220), an inverse quantizer (230), anoise generator (240), an inverse weighter (250), an inversemulti-channel transformer (260), and an inverse frequency transformer(270).

The DEMUX (210) parses information in the bitstream (205) and sendsinformation to the modules of the decoder (200). The DEMUX (210)includes one or more buffers to compensate for short-term variations inbitrate due to fluctuations in complexity of the audio, network jitter,and/or other factors.

The entropy decoder (220) losslessly decompresses entropy codes receivedfrom the DEMUX (210), producing quantized frequency coefficient data.The entropy decoder (220) typically applies the inverse of the entropyencoding technique used in the encoder.

The inverse quantizer (230) receives a quantization step size from theDEMUX (210) and receives quantized frequency coefficient data from theentropy decoder (220). The inverse quantizer (230) applies thequantization step size to the quantized frequency coefficient data topartially reconstruct the frequency coefficient data.

From the DEMUX (210), the noise generator (240) receives informationindicating which bands in a block of data are noise substituted as wellas any parameters for the form of the noise. The noise generator (240)generates the patterns for the indicated bands, and passes theinformation to the inverse weighter (250).

The inverse weighter (250) receives the weighting factors from the DEMUX(210), patterns for any noise-substituted bands from the noise generator(240), and the partially reconstructed frequency coefficient data fromthe inverse quantizer (230). As necessary, the inverse weighter (250)decompresses the weighting factors, for example, entropy decoding,inverse differentially coding, and inverse quantizing the elements ofthe quantization matrix. The inverse weighter (250) applies theweighting factors to the partially reconstructed frequency coefficientdata for bands that have not been noise substituted. The inverseweighter (250) then adds in the noise patterns received from the noisegenerator (240) for the noise-substituted bands.

The inverse multi-channel transformer (260) receives the reconstructedfrequency coefficient data from the inverse weighter (250) and channelmode information from the DEMUX (210). If multi-channel audio is inindependently coded channels, the inverse multi-channel transformer(260) passes the channels through. If multi-channel data is in jointlycoded channels, the inverse multi-channel transformer (260) converts thedata into independently coded channels.

The inverse frequency transformer (270) receives the frequencycoefficient data output by the multi-channel transformer (260) as wellas side information such as block sizes from the DEMUX (210). Theinverse frequency transformer (270) applies the inverse of the frequencytransform used in the encoder and outputs blocks of reconstructed audiosamples (295).

B. Disadvantages of Standard Perceptual Audio Encoders and Decoders

Although perceptual encoders and decoders as described above have goodoverall performance for many applications, they have several drawbacks,especially for compression and decompression of multi-channel audio. Thedrawbacks limit the quality of reconstructed multi-channel audio in somecases, for example, when the available bitrate is small relative to thenumber of input audio channels.

1. Inflexibility in Frame Partitioning for Multi-Channel Audio

In various respects, the frame partitioning performed by the encoder(100) of FIG. 1 is inflexible.

As previously noted, the frequency transformer (110) breaks a frame ofinput audio samples (105) into one or more overlapping windows forfrequency transformation, where larger windows provide better frequencyresolution and redundancy removal, and smaller windows provide bettertime resolution. The better time resolution helps control audiblepre-echo artifacts introduced when the signal transitions from lowenergy to high energy, but using smaller windows reducescompressibility, so the encoder must balance these considerations whenselecting window sizes. For multi-channel audio, the frequencytransformer (110) partitions the channels of a frame identically (i.e.,identical window configurations in the channels), which can beinefficient in some cases, as illustrated in FIGS. 3 a-3 c.

FIG. 3 a shows the waveforms (300) of an example stereo audio signal.The signal in channel 0 includes transient activity, whereas the signalin channel 1 is relatively stationary. The encoder (100) detects thesignal transition in channel 0 and, to reduce pre-echo, divides theframe into smaller overlapping, modulated windows (301) as shown in FIG.3 b. For the sake of simplicity, FIG. 3 c shows the overlapped windowconfiguration (302) in boxes, with dotted lines delimiting frameboundaries. Later figures also follow this convention.

A drawback of forcing all channels to have an identical windowconfiguration is that a stationary signal in one or more channels (e.g.,channel 1 in FIGS. 3 a- 3 c) may be broken into smaller windows,lowering coding gains. Alternatively, the encoder (100) might force allchannels to use larger windows, introducing pre-echo into one or morechannels that have transients. This problem is exacerbated when morethan two channels are to be coded.

AAC allows pair-wise grouping of channels for multi-channel transforms.Among left, right, center, back left, and back right channels, forexample, the left and right channels might be grouped for stereo coding,and the back left and back right channels might be grouped for stereocoding. Different groups can have different window configurations, butboth channels of a particular group have the same window configurationif stereo coding is used. This limits the flexibility of partitioningfor multi-channel transforms in the AAC system, as does the use of onlypair-wise groupings.

2. Inflexibility in Multi-Channel Transforms

The encoder (100) of FIG. 1 exploits some inter-channel redundancy, butis inflexible in various respects in terms of multi-channel transforms.The encoder (100) allows two kinds of transforms: (a) an identitytransform (which is equivalent to no transform at all) or (b)sum-difference coding of stereo pairs. These limitations constrainmulti-channel coding of more than two channels. Even in AAC, which canwork with more than two channels, a multi-channel transform is limitedto only a pair of channels at a time.

Several groups have experimented with multi-channel transformations forsurround sound channels. For example, see Yang et al., “An Inter-ChannelRedundancy Removal Approach for High-Quality Multichannel AudioCompression,” AES 109^(th) Convention, Los Angeles, September 2000[“Yang”], and Wang et al., “A Multichannel Audio Coding Algorithm forInter-Channel Redundancy Removal,” AES 110^(th) Convention, Amsterdam,Netherlands, May 2001 [“Wang”]. The Yang system uses a Karhunen-LoeveTransform [“KLT”] across channels to decorrelate the channels for goodcompression factors. The Wang system uses an integer-to-integer DiscreteCosine Transform [“DCT”]. Both systems give some good results, but stillhave several limitations.

First, using a KLT on audio samples (whether across the time domain orfrequency domain as in the Yang system) does not control the distortionintroduced in reconstruction. The KLT in the Yang system is not usedsuccessfully for perceptual audio coding of multi-channel audio. TheYang system does not control the amount of leakage from one (e.g.,heavily quantized) coded channel across to multiple reconstructedchannels in the inverse multi-channel transform. This shortcoming ispointed out in Kuo et al, “A Study of Why Cross Channel Prediction IsNot Applicable to Perceptual Audio Coding,” IEEE Signal Proc. Letters,vol. 8, no. 9, September 2001. In other words, quantization that is“inaudible” in one coded channel may become audible when spread inmultiple reconstructed channels, since inverse weighting is performedbefore the inverse multi-channel transform. The Wang system overcomesthis problem by placing the multi-channel transform after weighting andquantization in the encoder (and placing the inverse multi-channeltransform before inverse quantization and inverse weighting in thedecoder). The Wang system, however, has various other shortcomings.Performing the quantization prior to multi-channel transformation meansthat the multi-channel transformation must be integer-to-integer,limiting the number of transformations possible and limiting redundancyremoval across channels.

Second, the Yang system is limited to KLT transforms. While KLTtransforms adapt to the audio data being compressed, the flexibility ofthe Yang system to use different kinds of transforms is limited.Similarly, the Wang system uses integer-to-integer DCT for multi-channeltransforms, which is not as good as conventional DCTs in terms of energycompaction, and the flexibility of the Wang system to use differentkinds of transforms is limited.

Third, in the Yang and Wang systems, there is no mechanism to controlwhich channels get transformed together, nor is there a mechanism toselectively group different channels at different times formulti-channel transformation. Such control helps limit the leakage ofcontent across totally incompatible channels. Moreover, even channelsthat are compatible overall may be incompatible over some periods.

Fourth, in the Yang system, the multi-channel transformer lacks controlover whether to apply the multi-channel transform at the frequency bandlevel. Even among channels that are compatible overall, the channelsmight not be compatible at some frequencies or in some frequency bands.Similarly, the multi-channel transform of the encoder (100) of FIG. 1lacks control at the sub-channel level; it does not control which bandsof frequency coefficient data are multi-channel transformed, whichignores the inefficiencies that may result when less than all frequencybands of the input channels correlate.

Fifth, even when source channels are compatible, there is often a needto control the number of channels transformed together, so as to limitdata overflow and reduce memory accesses while implementing thetransform. In particular, the KLT of the Yang system is computationallycomplex. On the other hand, reducing the transform size also potentiallyreduces the coding gain compared to bigger transforms. Sixth, sendinginformation specifying multi-channel transformations can be costly interms of bitrate. This is particularly true for the KLT of the Yangsystem, as the transform coefficients for the covariance matrix sent arereal numbers.

Seventh, for low bitrate multi-channel audio, the quality of thereconstructed channels is very limited. Aside from the requirements ofcoding for low bitrate, this is in part due to the inability of thesystem to selectively and gracefully cut down the number of channels forwhich information is actually encoded.

3. Inefficiencies in Quantization and Weighting

In the encoder (100) of FIG. 1, the weighter (140) shapes distortionacross bands in audio data and the quantizer (150) sets quantizationstep sizes to change the amplitude of the distortion for a frame andthereby balance quality versus bitrate. While the encoder (100) achievesa good balance of quality and bitrate in most applications, the encoder(100) still has several drawbacks.

First, the encoder (100) lacks direct control over quality at thechannel level. The weighting factors shape overall distortion acrossquantization bands for an individual channel. The uniform, scalarquantization step size affects the amplitude of the distortion acrossall frequency bands and channels for a frame. Short of imposing veryhigh or very low quality on all channels, the encoder (100) lacks directcontrol over setting equal or at least comparable quality in thereconstructed output for all channels.

Second, when weighting factors are lossy compressed, the encoder (100)lacks control over the resolution of quantization of the weightingfactors. For direct compression of a quantization matrix, the encoder(100) uniformly quantizes elements of the quantization matrix, then usesdifferential coding and Huffman coding. The uniform quantization of maskelements does not adapt to changes in available bitrate or signalcomplexity. As a result, in some cases quantization matrices are encodedwith more resolution than is needed given the overall low quality of thereconstructed audio, and in other cases quantization matrices areencoded with less resolution than should be used given the high qualityof the reconstructed audio.

Third, the direct compression of quantization matrices in the encoder(100) fails to exploit temporal redundancies in the quantizationmatrices. The direct compression removes redundancy within a particularquantization matrix, but ignores temporal redundancy in a series ofquantization matrices.

C. Down-Mixing Audio Channels

Aside from multi-channel audio encoding and decoding, Dolby Pro-Logicand several other systems perform down-mixing of multi-channel audio tofacilitate compatibility with speaker configurations with differentnumbers of speakers. In the Dolby Pro-Logic down-mixing, for example,four channels are mixed down to two channels, with each of the twochannels having some combination of the audio data in the original fourchannels. The two channels can be output on stereo-channel equipment, orthe four channels can be reconstructed from the two-channels for outputon four-channel equipment.

While down-mixing of this nature solves some compatibility problems, itis limited to certain set configurations, for example, four to twochannel down-mixing. Moreover, the mixing formulas are pre-determinedand do not allow changes over time to adapt to the signal.

SUMMARY

In summary, the detailed description is directed to strategies forencoding and decoding multi-channel audio. For example, an audio encoderuses one or more techniques to improve the quality and/or bitrate ofmulti-channel audio data. This improves the overall listening experienceand makes computer systems a more compelling platform for creating,distributing, and playing back high-quality multi-channel audio. Theencoding and decoding strategies described herein include varioustechniques and tools, which can be used in combination or independently.

According to a first aspect of the strategies described herein, an audioencoder performs a pre-processing multi-channel transform onmulti-channel audio data. The encoder varies the transform during theencoding so as to control quality. For low bitrate coding, for example,the encoder alters or drops one or more of the original audio channelsso as to reduce coding complexity and improve the overall perceivedquality of the audio.

According to a second aspect of the strategies described herein, anaudio decoder performs a post-processing multi-channel transform ondecoded multi-channel audio data. The decoder uses the transform for anyof multiple different purposes. For example, the decoder optionallyre-matrixes time domain audio samples to create phantom channels atplayback or to perform special effects.

According to a third aspect of the strategies described herein, an audioencoder groups multiple windows from different channels into one or moretiles and outputs tile configuration information. For example, theencoder groups windows from different channels into a single tile whenthe windows have the same start time and the same stop time, whichallows the encoder to isolate transients that appear in a particularchannel with small windows (reducing pre-echo artifacts), but use largewindows for frequency resolution and temporal redundancy reduction inother channels.

According to a fourth aspect of the strategies described herein, anaudio encoder weights multi-channel audio data and then, after theweighting but before later quantization, performs a multi-channeltransform on the weighted audio data. This ordering can reduce leakageof audible quantization noise across channels upon reconstruction.

According to a fifth aspect of the strategies described herein, an audioencoder selectively groups multiple channels of audio data into multiplechannel groups for multi-channel transforms. The encoder groups themultiple channels differently at different times in an audio sequence.This can improve performance by giving the encoder more precise controlover application of multi-channel transforms to relatively correlatedparts of the data.

According to a sixth aspect of the strategies described herein, an audioencoder selectively turns a selected transform on/off at multiplefrequency bands. For example, the encoder selectively excludes bandsthat are not compatible in multi-channel transforms, which again givesthe encoder more precise control over application of multi-channeltransforms to relatively correlated parts of the data.

According to a seventh aspect of the strategies described herein, anaudio encoder transforms multi-channel audio data according to ahierarchy of multi-channel transforms in multiple stages. For example,the hierarchy emulates another transform while reducing computationcomplexity compared to the other transform.

According to a eighth aspect of the strategies described herein, anaudio encoder selects a multi-channel transform from among multipleavailable types of multi-channel transforms. For example, the typesinclude multiple pre-defined transforms as well as a custom transform.In this way, the encoder reduces the bitrate used to specify transforms.

According to a ninth aspect of the strategies described herein, an audioencoder computes an arbitrary unitary transform matrix then factorizesit. The encoder performs the factorized transform and outputsinformation for it. In this way, the encoder efficiently compresseseffective multi-channel transform matrices.

For several of the aspects described above in terms of an audio encoder,an audio decoder performs corresponding processing and decoding.

The various features and advantages of the invention will be madeapparent from the following detailed description of embodiments thatproceeds with reference to the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an audio encoder according to the priorart.

FIG. 2 is a block diagram of an audio decoder according to the priorart.

FIGS. 3 a-3 c are charts showing window configurations for a frame ofstereo audio data according to the prior art.

FIG. 4 is a chart showing six channels in a 5.1 channel/speakerconfiguration.

FIG. 5 is a block diagram of a suitable computing environment in whichdescribed embodiments may be implemented.

FIG. 6 is a block diagram of an audio encoder in which describedembodiments may be implemented.

FIG. 7 is a block diagram of an audio decoder in which describedembodiments may be implemented.

FIG. 8 is a flowchart showing a generalized technique for multi-channelpre-processing.

FIGS. 9 a-9 e are charts showing example matrices for multi-channelpre-processing.

FIG. 10 is a flowchart showing a technique for multi-channelpre-processing in which the transform matrix potentially changes on aframe-by-frame basis.

FIGS. 11 a and 11 b are charts showing example tile configurations formulti-channel audio.

FIG. 12 is a flowchart showing a generalized technique for configuringtiles of multi-channel audio.

FIG. 13 is a flowchart showing a technique for concurrently configuringtiles and sending tile information for multi-channel audio according toa particular bitstream syntax.

FIG. 14 is a flowchart showing a generalized technique for performing amulti-channel transform after perceptual weighting.

FIG. 15 is a flowchart showing a generalized technique for performing aninverse multi-channel transform before inverse perceptual weighting.

FIG. 16 is a flowchart showing a technique for grouping channels in atile for multi-channel transformation in one implementation.

FIG. 17 is a flowchart showing a technique for retrieving channel groupinformation and multi-channel transform information for a tile from abitstream according to a particular bitstream syntax.

FIG. 18 is a flowchart showing a technique for selectively includingfrequency bands of a channel group in a multi-channel transform in oneimplementation.

FIG. 19 is a flowchart showing a technique for retrieving band on/offinformation for a multi-channel transform for a channel group of a tilefrom a bitstream according to a particular bitstream syntax.

FIG. 20 is a flowchart showing a generalized technique for emulating amulti-channel transform using a hierarchy of simpler multi-channeltransforms.

FIG. 21 is a chart showing an example hierarchy of multi-channeltransforms.

FIG. 22 is a flowchart showing a technique for retrieving informationfor a hierarchy of multi-channel transforms for channel groups from abitstream according to a particular bitstream syntax.

FIG. 23 is a flowchart showing a generalized technique for selecting amulti-channel transform type from among plural available types.

FIG. 24 is a flowchart showing a generalized technique for retrieving amulti-channel transform type from among plural available types andperforming an inverse multi-channel transform.

FIG. 25 is a flowchart showing a technique for retrieving multi-channeltransform information for a channel group from a bitstream according toa particular bitstream syntax.

FIG. 26 is a chart showing the general form of a rotation matrix forGivens rotations for representing a multi-channel transform matrix.

FIGS. 27 a-27 c are charts showing example rotation matrices for Givensrotations for representing a multi-channel transform matrix.

FIG. 28 is a flowchart showing a generalized technique for representinga multi-channel transform matrix using quantized Givens factorizingrotations.

FIG. 29 is a flowchart showing a technique for retrieving informationfor a generic unitary transform for a channel group from a bitstreamaccording to a particular bitstream syntax.

FIG. 30 is a flowchart showing a technique for retrieving an overalltile quantization factor for a tile from a bitstream according to aparticular bitstream syntax.

FIG. 31 is a flowchart showing a generalized technique for computingper-channel quantization step modifiers for multi-channel audio data.

FIG. 32 is a flowchart showing a technique for retrieving per-channelquantization step modifiers from a bitstream according to a particularbitstream syntax.

FIG. 33 is a flowchart showing a generalized technique for adaptivelysetting a quantization step size for quantization matrix elements.

FIG. 34 is a flowchart showing a generalized technique for retrieving anadaptive quantization step size for quantization matrix elements.

FIGS. 35 and 36 are flowcharts showing techniques for compressingquantization matrices using temporal prediction.

FIG. 37 is a chart showing a mapping of bands for prediction ofquantization matrix elements.

FIG. 38 is a flowchart showing a technique for retrieving and decodingquantization matrices compressed using temporal prediction according toa particular bitstream syntax.

FIG. 39 is a flowchart showing a generalized technique for multi-channelpost-processing.

FIG. 40 is a chart showing an example matrix for multi-channelpost-processing.

FIG. 41 is a flowchart showing a technique for multi-channelpost-processing in which the transform matrix potentially changes on aframe-by-frame basis.

FIG. 42 is a flowchart showing a technique for identifying andretrieving a transform matrix for multi-channel post-processingaccording to a particular bitstream syntax.

DETAILED DESCRIPTION

Described embodiments of the present invention are directed totechniques and tools for processing audio information in encoding anddecoding. In described embodiments, an audio encoder uses severaltechniques to process audio during encoding. An audio decoder usesseveral techniques to process audio during decoding. While thetechniques are described in places herein as part of a single,integrated system, the techniques can be applied separately, potentiallyin combination with other techniques. In alternative embodiments, anaudio processing tool other than an encoder or decoder implements one ormore of the techniques.

In some embodiments, an encoder performs multi-channel pre-processing.For low bitrate coding, for example, the encoder optionally re-matrixestime domain audio samples to artificially increase inter-channelcorrelation. This makes subsequent compression of the affected channelsmore efficient by reducing coding complexity. The pre-processingdecreases channel separation, but can improve overall quality.

In some embodiments, an encoder and decoder work with multi-channelaudio configured into tiles of windows. For example, the encoderpartitions frames of multi-channel audio on a per-channel basis, suchthat each channel can have a window configuration independent of theother channels. The encoder then groups windows of the partitionedchannels into tiles for multi-channel transformations. This allows theencoder to isolate transients that appear in a particular channel of aframe with small windows (reducing pre-echo artifacts), but use largewindows for frequency resolution and temporal redundancy reduction inother channels of the frame.

In some embodiments, an encoder performs one or more flexiblemulti-channel transform techniques. A decoder performs the correspondinginverse multi-channel transform techniques. In first techniques, theencoder performs a multi-channel transform after perceptual weighting inthe encoder, which reduces leakage of audible quantization noise acrosschannels upon reconstruction. In second techniques, an encoder flexiblygroups channels for multi-channel transforms to selectively includechannels at different times. In third techniques, an encoder flexiblyincludes or excludes particular frequencies bands in multi-channeltransforms, so as to selectively include compatible bands. In fourthtechniques, an encoder reduces the bitrate associated with transformmatrices by selectively using pre-defined matrices or using Givensrotations to parameterize custom transform matrices. In fifthtechniques, an encoder performs flexible hierarchical multi-channeltransforms.

In some embodiments, an encoder performs one or more improvedquantization or weighting techniques. A corresponding decoder performsthe corresponding inverse quantization or inverse weighting techniques.In first techniques, an encoder computes and applies per-channelquantization step modifiers, which gives the encoder more control overbalancing reconstruction quality between channels. In second techniques,an encoder uses a flexible quantization step size for quantizationmatrix elements, which allows the encoder to change the resolution ofthe elements of quantization matrices. In third techniques, an encoderuses temporal prediction in compression of quantization matrices toreduce bitrate.

In some embodiments, a decoder performs multi-channel post-processing.For example, the decoder optionally re-matrixes time domain audiosamples to create phantom channels at playback, perform special effects,fold down channels for playback on fewer speakers, or for any otherpurpose.

In the described embodiments, multi-channel audio includes six channelsof a standard 5.1 channel/speaker configuration as shown in the matrix(400) of FIG. 4. The “5” channels are the left, right, center, backleft, and back right channels, and are conventionally spatially orientedfor surround sound. The “1” channel is the sub-woofer or low-frequencyeffects channel. For the sake of clarity, the order of the channelsshown in the matrix (400) is also used for matrices and equations in therest of the specification. Alternative embodiments use multi-channelaudio having a different ordering, number (e.g., 7.1, 9.1, 2), and/orconfiguration of channels.

In described embodiments, the audio encoder and decoder perform varioustechniques. Although the operations for these techniques are typicallydescribed in a particular, sequential order for the sake ofpresentation, it should be understood that this manner of descriptionencompasses minor rearrangements in the order of operations, unless aparticular ordering is required. For example, operations describedsequentially may in some cases be rearranged or performed concurrently.Moreover, for the sake of simplicity, flowcharts typically do not showthe various ways in which particular techniques can be used inconjunction with other techniques.

I. Computing Environment

FIG. 5 illustrates a generalized example of a suitable computingenvironment (500) in which described embodiments may be implemented. Thecomputing environment (500) is not intended to suggest any limitation asto scope of use or functionality of the invention, as the presentinvention may be implemented in diverse general-purpose orspecial-purpose computing environments.

With reference to FIG. 5, the computing environment (500) includes atleast one processing unit (510) and memory (520). In FIG. 5, this mostbasic configuration (530) is included within a dashed line. Theprocessing unit (510) executes computer-executable instructions and maybe a real or a virtual processor. In a multi-processing system, multipleprocessing units execute computer-executable instructions to increaseprocessing power. The memory (520) may be volatile memory (e.g.,registers, cache, RAM), non-volatile memory (e.g., ROM, EEPROM, flashmemory, etc.), or some combination of the two. The memory (520) storessoftware (580) implementing audio processing techniques according to oneor more of the described embodiments.

A computing environment may have additional features. For example, thecomputing environment (500) includes storage (540), one or more inputdevices (550), one or more output devices (560), and one or morecommunication connections (570). An interconnection mechanism (notshown) such as a bus, controller, or network interconnects thecomponents of the computing environment (500). Typically, operatingsystem software (not shown) provides an operating environment for othersoftware executing in the computing environment (500), and coordinatesactivities of the components of the computing environment (500).

The storage (540) may be removable or non-removable, and includesmagnetic disks, magnetic tapes or cassettes, CD-ROMs, CD-RWs, DVDs, orany other medium which can be used to store information and which can beaccessed within the computing environment (500). The storage (540)stores instructions for the software (580) implementing audio processingtechniques according to one or more of the described embodiments.

The input device(s) (550) may be a touch input device such as akeyboard, mouse, pen, or trackball, a voice input device, a scanningdevice, network adapter, or another device that provides input to thecomputing environment (500). For audio, the input device(s) (550) may bea sound card or similar device that accepts audio input in analog ordigital form, or a CD-ROM/DVD reader that provides audio samples to thecomputing environment. The output device(s) (560) may be a display,printer, speaker, CD/DVD-writer, network adapter, or another device thatprovides output from the computing environment (500).

The communication connection(s) (570) enable communication over acommunication medium to another computing entity. The communicationmedium conveys information such as computer-executable instructions,compressed audio information, or other data in a modulated data signal.A modulated data signal is a signal that has one or more of itscharacteristics set or changed in such a manner as to encode informationin the signal. By way of example, and not limitation, communicationmedia include wired or wireless techniques implemented with anelectrical, optical, RF, infrared, acoustic, or other carrier.

The invention can be described in the general context ofcomputer-readable media. Computer-readable media are any available mediathat can be accessed within a computing environment. By way of example,and not limitation, with the computing environment (500),computer-readable media include memory (520), storage (540),communication media, and combinations of any of the above.

The invention can be described in the general context ofcomputer-executable instructions, such as those included in programmodules, being executed in a computing environment on a target real orvirtual processor. Generally, program modules include routines,programs, libraries, objects, classes, components, data structures, etc.that perform particular tasks or implement particular abstract datatypes. The functionality of the program modules may be combined or splitbetween program modules as desired in various embodiments.Computer-executable instructions for program modules may be executedwithin a local or distributed computing environment.

For the sake of presentation, the detailed description uses terms like“determine,” “generate,” “adjust,” and “apply” to describe computeroperations in a computing environment. These terms are high-levelabstractions for operations performed by a computer, and should not beconfused with acts performed by a human being. The actual computeroperations corresponding to these terms vary depending onimplementation.

II. Generalized Audio Encoder and Decoder

FIG. 6 is a block diagram of a generalized audio encoder (600) in whichdescribed embodiments may be implemented. FIG. 7 is a block diagram of ageneralized audio decoder (700) in which described embodiments may beimplemented.

The relationships shown between modules within the encoder and decoderindicate flows of information in the encoder and decoder; otherrelationships are not shown for the sake of simplicity. Depending onimplementation and the type of compression desired, modules of theencoder or decoder can be added, omitted, split into multiple modules,combined with other modules, and/or replaced with like modules. Inalternative embodiments, encoders or decoders with different modulesand/or other configurations process audio data.

A. Generalized Audio Encoder

The generalized audio encoder (600) includes a selector (608), amulti-channel pre-processor (610), a partitioner/tile configurer (620),a frequency transformer (630), a perception modeler (640), aquantization band weighter (642), a channel weighter (644), amulti-channel transformer (650), a quantizer (660), an entropy encoder(670), a controller (680), a mixed/pure lossless coder (672) andassociated entropy encoder (674), and a bitstream multiplexer [“MUX”](690).

The encoder (600) receives a time series of input audio samples (605) atsome sampling depth and rate in pulse code modulated [“PCM”] format. Formost of the described embodiments, the input audio samples (605) are formulti-channel audio (e.g., stereo, surround), but the input audiosamples (605) can instead be mono. The encoder (600) compresses theaudio samples (605) and multiplexes information produced by the variousmodules of the encoder (600) to output a bitstream (695) in a formatsuch as a Windows Media Audio [“WMA”] format or Advanced StreamingFormat [“ASF”]. Alternatively, the encoder (600) works with other inputand/or output formats.

The selector (608) selects between multiple encoding modes for the audiosamples (605). In FIG. 6, the selector (608) switches between amixed/pure lossless coding mode and a lossy coding mode. The losslesscoding mode includes the mixed/pure lossless coder (672) and istypically used for high quality (and high bitrate) compression. Thelossy coding mode includes components such as the weighter (642) andquantizer (660) and is typically used for adjustable quality (andcontrolled bitrate) compression. The selection decision at the selector(608) depends upon user input or other criteria. In certaincircumstances (e.g., when lossy compression fails to deliver adequatequality or overproduces bits), the encoder (600) may switch from lossycoding over to mixed/pure lossless coding for a frame or set of frames.

For lossy coding of multi-channel audio data, the multi-channelpre-processor (610) optionally re-matrixes the time-domain audio samples(605). In some embodiments, the multi-channel pre-processor (610)selectively re-matrixes the audio samples (605) to drop one or morecoded channels or increase inter-channel correlation in the encoder(600), yet allow reconstruction (in some form) in the decoder (700).This gives the encoder additional control over quality at the channellevel. The multi-channel pre-processor (610) may send side informationsuch as instructions for multi-channel post-processing to the MUX (690).For additional detail about the operation of the multi-channelpre-processor in some embodiments, see the section entitled“Multi-Channel Pre-Processing.” Alternatively, the encoder (600)performs another form of multi-channel pre-processing.

The partitioner/tile configurer (620) partitions a frame of audio inputsamples (605) into sub-frame blocks (i.e., windows) with time-varyingsize and window shaping functions. The sizes and windows for thesub-frame blocks depend upon detection of transient signals in theframe, coding mode, as well as other factors.

If the encoder (600) switches from lossy coding to mixed/pure losslesscoding, sub-frame blocks need not overlap or have a windowing functionin theory (i.e., non-overlapping, rectangular-window blocks), buttransitions between lossy coded frames and other frames may requirespecial treatment. The partitioner/tile configurer (620) outputs blocksof partitioned data to the mixed/pure lossless coder (672) and outputsside information such as block sizes to the MUX (690). For additionaldetail about partitioning and windowing for mixed or pure losslesslycoded frames, see the related application entitled “Unified Lossy andLossless Audio Compression.”

When the encoder (600) uses lossy coding, variable-size windows allowvariable temporal resolution. Small blocks allow for greaterpreservation of time detail at short but active transition segments.Large blocks have better frequency resolution and worse time resolution,and usually allow for greater compression efficiency at longer and lessactive segments, in part because frame header and side information isproportionally less than in small blocks, and in part because it allowsfor better redundancy removal. Blocks can overlap to reduce perceptiblediscontinuities between blocks that could otherwise be introduced bylater quantization. The partitioner/tile configurer (620) outputs blocksof partitioned data to the frequency transformer (630) and outputs sideinformation such as block sizes to the MUX (690). For additionalinformation about transient detection and partitioning criteria in someembodiments, see U.S. patent application Ser. No. 10/016,918, entitled“Adaptive Window-Size Selection in Transform Coding,” filed Dec. 14,2001, hereby incorporated by reference. Alternatively, thepartitioner/tile configurer (620) uses other partitioning criteria orblock sizes when partitioning a frame into windows.

In some embodiments, the partitioner/tile configurer (620) partitionsframes of multi-channel audio on a per-channel basis. Thepartitioner/tile configurer (620) independently partitions each channelin the frame, if quality/bitrate allows. This allows, for example, thepartitioner/tile configurer (620) to isolate transients that appear in aparticular channel with smaller windows, but use larger windows forfrequency resolution or compression efficiency in other channels. Thiscan improve compression efficiency by isolating transients on a perchannel basis, but additional information specifying the partitions inindividual channels is needed in many cases. Windows of the same sizethat are co-located in time may qualify for further redundancy reductionthrough multi-channel transformation. Thus, the partitioner/tileconfigurer (620) groups windows of the same size that are co-located intime as a tile. For additional detail about tiling in some embodiments,see the section entitled “Tile Configuration.”

The frequency transformer (630) receives audio samples and converts theminto data in the frequency domain. The frequency transformer (630)outputs blocks of frequency coefficient data to the weighter (642) andoutputs side information such as block sizes to the MUX (690). Thefrequency transformer (630) outputs both the frequency coefficients andthe side information to the perception modeler (640). In someembodiments, the frequency transformer (630) applies a time-varyingModulated Lapped Transform [“MLT”] to the sub-frame blocks, whichoperates like a DCT modulated by the sine window function(s) of thesub-frame blocks. Alternative embodiments use other varieties of MLT, ora DCT or other type of modulated or non-modulated, overlapped ornon-overlapped frequency transform, or use subband or wavelet coding.

The perception modeler (640) models properties of the human auditorysystem to improve the perceived quality of the reconstructed audiosignal for a given bitrate. Generally, the perception modeler (640)processes the audio data according to an auditory model, then providesinformation to the weighter (642) which can be used to generateweighting factors for the audio data. The perception modeler (640) usesany of various auditory models and passes excitation pattern informationor other information to the weighter (642).

The quantization band weighter (642) generates weighting factors forquantization matrices based upon the information received from theperception modeler (640) and applies the weighting factors to the datareceived from the frequency transformer (630). The weighting factors fora quantization matrix include a weight for each of multiple quantizationbands in the audio data. The quantization bands can be the same ordifferent in number or position from the critical bands used elsewherein the encoder (600), and the weighting factors can vary in amplitudesand number of quantization bands from block to block. The quantizationband weighter (642) outputs weighted blocks of coefficient data to thechannel weighter (644) and outputs side information such as the set ofweighting factors to the MUX (690). The set of weighting factors can becompressed for more efficient representation. If the weighting factorsare lossy compressed, the reconstructed weighting factors are typicallyused to weight the blocks of coefficient data. For additional detailabout computation and compression of weighting factors in someembodiments, see the section entitled “Quantization and Weighting.”Alternatively, the encoder (600) uses another form of weighting or skipsweighting.

The channel weighter (644) generates channel-specific weight factors(which are scalars) for channels based on the information received fromthe perception modeler (640) and also on the quality of locallyreconstructed signal. The scalar weights (also called quantization stepmodifiers) allow the encoder (600) to give the reconstructed channelsapproximately uniform quality. The channel weight factors can vary inamplitudes from channel to channel and block to block, or at some otherlevel. The channel weighter (644) outputs weighted blocks of coefficientdata to the multi-channel transformer (650) and outputs side informationsuch as the set of channel weight factors to the MUX (690). The channelweighter (644) and quantization band weighter (642) in the flow diagramcan be swapped or combined together. For additional detail aboutcomputation and compression of weighting factors in some embodiments,see the section entitled “Quantization and Weighting.” Alternatively,the encoder (600) uses another form of weighting or skips weighting.

For multi-channel audio data, the multiple channels of noise-shapedfrequency coefficient data produced by the channel weighter (644) oftencorrelate, so the multi-channel transformer (650) may apply amulti-channel transform. For example, the multi-channel transformer(650) selectively and flexibly applies the multi-channel transform tosome but not all of the channels and/or quantization bands in the tile.This gives the multi-channel transformer (650) more precise control overapplication of the transform to relatively correlated parts of the tile.To reduce computational complexity, the multi-channel transformer (650)may use a hierarchical transform rather than a one-level transform. Toreduce the bitrate associated with the transform matrix, themulti-channel transformer (650) selectively uses pre-defined matrices(e.g., identity/no transform, Hadamard, DCT Type II) or custom matrices,and applies efficient compression to the custom matrices. Finally, sincethe multi-channel transform is downstream from the weighter (642), theperceptibility of noise (e.g., due to subsequent quantization) thatleaks between channels after the inverse multi-channel transform in thedecoder (700) is controlled by inverse weighting. For additional detailabout multi-channel transforms in some embodiments, see the sectionentitled “Flexible Multi-Channel Transforms.” Alternatively, the encoder(600) uses other forms of multi-channel transforms or no transforms atall. The multi-channel transformer (650) produces side information tothe MUX (690) indicating, for example, the multi-channel transforms usedand multi-channel transformed parts of tiles.

The quantizer (660) quantizes the output of the multi-channeltransformer (650), producing quantized coefficient data to the entropyencoder (670) and side information including quantization step sizes tothe MUX (690). In FIG. 6, the quantizer (660) is an adaptive, uniform,scalar quantizer that computes a quantization factor per tile. The tilequantization factor can change from one iteration of a quantization loopto the next to affect the bitrate of the entropy encoder (660) output,and the per-channel quantization step modifiers can be used to balancereconstruction quality between channels. For additional detail aboutquantization in some embodiments, see the section entitled “Quantizationand Weighting.” In alternative embodiments, the quantizer is anon-uniform quantizer, a vector quantizer, and/or a non-adaptivequantizer, or uses a different form of adaptive, uniform, scalarquantization. In other alternative embodiments, the quantizer (660),quantization band weighter (642), channel weighter (644), andmulti-channel transformer (650) are fused and the fused moduledetermines various weights all at once.

The entropy encoder (670) losslessly compresses quantized coefficientdata received from the quantizer (660). In some embodiments, the entropyencoder (670) uses adaptive entropy encoding as described in the relatedapplication entitled, “Entropy Coding by Adapting Coding Between Leveland Run Length/Level Modes.” Alternatively, the entropy encoder (670)uses some other form or combination of multi-level run length coding,variable-to-variable length coding, run length coding, Huffman coding,dictionary coding, arithmetic coding, LZ coding, or some other entropyencoding technique. The entropy encoder (670) can compute the number ofbits spent encoding audio information and pass this information to therate/quality controller (680).

The controller (680) works with the quantizer (660) to regulate thebitrate and/or quality of the output of the encoder (600). Thecontroller (680) receives information from other modules of the encoder(600) and processes the received information to determine desiredquantization factors given current conditions. The controller (670)outputs the quantization factors to the quantizer (660) with the goal ofsatisfying quality and/or bitrate constraints.

The mixed/pure lossless encoder (672) and associated entropy encoder(674) compress audio data for the mixed/pure lossless coding mode. Theencoder (600) uses the mixed/pure lossless coding mode for an entiresequence or switches between coding modes on a frame-by-frame,block-by-block, tile-by-tile, or other basis. For additional detailabout the mixed/pure lossless coding mode, see the related applicationentitled “Unified Lossy and Lossless Audio Compression.” Alternatively,the encoder (600) uses other techniques for mixed and/or pure losslessencoding.

The MUX (690) multiplexes the side information received from the othermodules of the audio encoder (600) along with the entropy encoded datareceived from the entropy encoders (670, 674). The MUX (690) outputs theinformation in a WMA format or another format that an audio decoderrecognizes. The MUX (690) includes a virtual buffer that stores thebitstream (695) to be output by the encoder (600). The virtual bufferthen outputs data at a relatively constant bitrate, while quality maychange due to complexity changes in the input. The current fullness andother characteristics of the buffer can be used by the controller (680)to regulate quality and/or bitrate. Alternatively, the output bitratecan vary over time, and the quality is kept relatively constant. Or, theoutput bitrate is only constrained to be less than a particular bitrate,which is either constant or time varying.

B. Generalized Audio Decoder

With reference to FIG. 7, the generalized audio decoder (700) includes abitstream demultiplexer [“DEMUX”] (710), one or more entropy decoders(720), a mixed/pure lossless decoder (722), a tile configuration decoder(730), an inverse multi-channel transformer (740), a inversequantizer/weighter (750), an inverse frequency transformer (760), anoverlapper/adder (770), and a multi-channel post-processor (780). Thedecoder (700) is somewhat simpler than the encoder (700) because thedecoder (700) does not include modules for rate/quality control orperception modeling.

The decoder (700) receives a bitstream (705) of compressed audioinformation in a WMA format or another format. The bitstream (705)includes entropy encoded data as well as side information from which thedecoder (700) reconstructs audio samples (795).

The DEMUX (710) parses information in the bitstream (705) and sendsinformation to the modules of the decoder (700). The DEMUX (710)includes one or more buffers to compensate for short-term variations inbitrate due to fluctuations in complexity of the audio, network jitter,and/or other factors.

The one or more entropy decoders (720) losslessly decompress entropycodes received from the DEMUX (710). The entropy decoder (720) typicallyapplies the inverse of the entropy encoding technique used in theencoder (600). For the sake of simplicity, one entropy decoder module isshown in FIG. 7, although different entropy decoders may be used forlossy and lossless coding modes, or even within modes. Also, for thesake of simplicity, FIG. 7 does not show mode selection logic. Whendecoding data compressed in lossy coding mode, the entropy decoder (720)produces quantized frequency coefficient data.

The mixed/pure lossless decoder (722) and associated entropy decoder(s)(720) decompress losslessly encoded audio data for the mixed/purelossless coding mode. For additional detail about decompression for themixed/pure lossless decoding mode, see the related application entitled“Unified Lossy and Lossless Audio Compression.” Alternatively, decoder(700) uses other techniques for mixed and/or pure lossless decoding.

The tile configuration decoder (730) receives and, if necessary, decodesinformation indicating the patterns of tiles for frames from the DEMUX(790). The tile pattern information may be entropy encoded or otherwiseparameterized. The tile configuration decoder (730) then passes tilepattern information to various other modules of the decoder (700). Foradditional detail about tile configuration decoding in some embodiments,see the section entitled “Tile Configuration.” Alternatively, thedecoder (700) uses other techniques to parameterize window patterns inframes.

The inverse multi-channel transformer (740) receives the quantizedfrequency coefficient data from the entropy decoder (720) as well astile pattern information from the tile configuration decoder (730) andside information from the DEMUX (710) indicating, for example, themulti-channel transform used and transformed parts of tiles. Using thisinformation, the inverse multi-channel transformer (740) decompressesthe transform matrix as necessary, and selectively and flexibly appliesone or more inverse multi-channel transforms to the audio data. Theplacement of the inverse multi-channel transformer (740) relative to theinverse quantizer/weighter (750) helps shape quantization noise that mayleak across channels. For additional detail about inverse multi-channeltransforms in some embodiments, see the section entitled “FlexibleMulti-Channel Transforms.”

The inverse quantizer/weighter (750) receives tile and channelquantization factors as well as quantization matrices from the DEMUX(710) and receives quantized frequency coefficient data from the inversemulti-channel transformer (740). The inverse quantizer/weighter (750)decompresses the received quantization factor/matrix information asnecessary, then performs the inverse quantization and weighting. Foradditional detail about inverse quantization and weighting in someembodiments, see the section entitled “Quantization and Weighting. Inalternative embodiments, the inverse quantizer/weighter applies theinverse of some other quantization techniques used in the encoder.

The inverse frequency transformer (760) receives the frequencycoefficient data output by the inverse quantizer/weighter (750) as wellas side information from the DEMUX (710) and tile pattern informationfrom the tile configuration decoder (730). The inverse frequencytransformer (770) applies the inverse of the frequency transform used inthe encoder and outputs blocks to the overlapper/adder (770).

In addition to receiving tile pattern information from the tileconfiguration decoder (730), the overlapper/adder (770) receives decodedinformation from the inverse frequency transformer (760) and/ormixed/pure lossless decoder (722). The overlapper/adder (770) overlapsand adds audio data as necessary and interleaves frames or othersequences of audio data encoded with different modes. For additionaldetail about overlapping, adding, and interleaving mixed or purelosslessly coded frames, see the related application entitled “UnifiedLossy and Lossless Audio Compression.” Alternatively, the decoder (700)uses other techniques for overlapping, adding, and interleaving frames.

The multi-channel post-processor (780) optionally re-matrixes thetime-domain audio samples output by the overlapper/adder (770). Themulti-channel post-processor selectively re-matrixes audio data tocreate phantom channels for playback, perform special effects such asspatial rotation of channels among speakers, fold down channels forplayback on fewer speakers, or for any other purpose. Forbitstream-controlled post-processing, the post-processing transformmatrices vary over time and are signaled or included in the bitstream(705). For additional detail about the operation of the multi-channelpost-processor in some embodiments, see the section entitled“Multi-Channel Post-Processing.” Alternatively, the decoder (700)performs another form of multi-channel post-processing.

III. Multi-Channel Pre-Processing

In some embodiments, an encoder such as the encoder (600) of FIG. 6performs multi-channel pre-processing on input audio samples in thetime-domain.

In general, when there are N source audio channels as input, the numberof coded channels produced by the encoder is also N. The coded channelsmay correspond one-to-one with the source channels, or the codedchannels may be multi-channel transform-coded channels. When the codingcomplexity of the source makes compression difficult or when the encoderbuffer is full, however, the encoder may alter or drop (i.e., not code)one or more of the original input audio channels. This can be done toreduce coding complexity and improve the overall perceived quality ofthe audio. For quality-driven pre-processing, the encoder performs themulti-channel pre-processing in reaction to measured audio quality so asto smoothly control overall audio quality and channel separation.

For example, the encoder may alter the multi-channel audio image to makeone or more channels less critical so that the channels are dropped atthe encoder yet reconstructed at the decoder as “phantom” channels.Outright deletion of channels can have a dramatic effect on quality, soit is done only when coding complexity is very high or the buffer is sofull that good quality reproduction cannot be achieved through othermeans.

The encoder can indicate to the decoder what action to take when thenumber of coded channels is less than the number of channels for output.Then, a multi-channel post-processing transform can be used in thedecoder to create phantom channels, as described below in the sectionentitled “Multi-Channel Post-Processing.” Or, the encoder can signal tothe decoder to perform multi-channel post-processing for anotherpurpose.

FIG. 8 shows a generalized technique (800) for multi-channelpre-processing. The encoder performs (810) multi-channel pre-processingon time-domain multi-channel audio data (805), producing transformedaudio data (815) in the time domain. For example, the pre-processinginvolves a general N to N transform, where N is the number of channels.The encoder multiplies N samples with a matrix A.y _(pre) =A _(pre) ·x _(pre)  (4),where x_(pre) and y_(pre) are the N channel input to and the output fromthe pre-processing, and A_(pre) is a general N×N transform matrix withreal (i.e., continuous) valued elements. The matrix A_(pre) can bechosen to artificially increase the inter-channel correlation in y_(pre)compared to x_(pre). This reduces complexity for the rest of theencoder, but at the cost of lost channel separation.

The output y_(pre) is then fed to the rest of the encoder, which encodes(820) the data using techniques shown in FIG. 6 or other compressiontechniques, producing encoded multi-channel audio data (825).

The syntax used by the encoder and decoder allows description of generalor pre-defined post-processing multi-channel transform matrices, whichcan vary or be turned on/off on a frame-to-frame basis. The encoder usesthis flexibility to limit stereo/surround image impairments, trading offchannel separation for better overall quality in certain circumstancesby artificially increasing inter-channel correlation. Alternatively, thedecoder and encoder use another syntax for multi-channel pre- andpost-processing, for example, one that allows changes in transformmatrices on a basis other than frame-to-frame.

FIGS. 9 a-9 e show multi-channel pre-processing transform matrices(900-904) used to artificially increase inter-channel correlation undercertain circumstances in the encoder. The encoder switches betweenpre-processing matrices to change how much inter-channel correlation isartificially increased between the left, right, and center channels, andbetween the back left and back right channels, in a 5.1 channel playbackenvironment.

In one implementation, at low bitrates, the encoder evaluates thequality of reconstructed audio over some period of time and, dependingon the result, selects one of the pre-processing matrices. The qualitymeasure evaluated by the encoder is Noise to Excitation Ratio [“NER”],which is the ratio of the energy in the noise pattern for areconstructed audio clip to the energy in the original digital audioclip. Low NER values indicate good quality, and high NER values indicatepoor quality. The encoder evaluates the NER for one or more previouslyencoded frames. For additional information about NER and other qualitymeasures, see U.S. patent application Ser. No. 10/017,861, entitled“Techniques for Measurement of Perceptual Audio Quality,” filed Dec. 14,2001, hereby incorporated by reference. Alternatively, the encoder usesanother quality measure, buffer fullness, and/or some other criteria toselect a pre-processing transform matrix, or the encoder evaluates adifferent period of multi-channel audio.

Returning to the examples shown in FIGS. 9 a-9 e, at low bitrates, theencoder slowly changes the pre-processing transform matrix based on theNER n of a particular stretch of audio clip. The encoder compares thevalue of n to threshold values n_(low) and n_(high), which areimplementation-dependent. In one implementation, n_(low) and n_(high)have the pre-determined values n_(low)=0.05 and n_(high)=0.1.Alternatively, n_(low) and n_(high) have different values or values thatchange over time in reaction to bitrate or other criteria, or theencoder switches between a different number of matrices.

A low value of n (e.g., n≦n_(low)) indicates good quality coding. So,the encoder uses the identity matrix A_(low) (900) shown in FIG. 9 a,effectively turning off the pre-processing.

On the other hand, a high value of n (e.g., n≧n_(high)) indicates poorquality coding. So, the encoder uses the matrix A_(high,1), (902) shownin FIG. 9 c. The matrix A_(high,1) (902) introduces severe surroundimage distortion, but at the same time imposes very high correlationbetween the left, right, and center channels, which improves subsequentcoding efficiency by reducing complexity. The multi-channel transformedcenter channel is the average of the original left, right, and centerchannels. The matrix A_(high,1) (902) also compromises the channelseparation between the rear channels—the input back left and back rightchannels are averaged.

An intermediate value of n (e.g., n_(low)<n<n_(high)) indicatesintermediate quality coding. So, the encoder may use the intermediatematrix A_(int er,1) (901) shown in FIG. 9 b. In the intermediate matrixA_(int er,1) (901), the factor a measures the relative position of nbetween n_(low) and n_(high).

$\begin{matrix}{\alpha = {\frac{n - n_{low}}{n_{high} - n_{low}}.}} & (5)\end{matrix}$The intermediate matrix A_(int er,1) (901) gradually transitions fromthe identity matrix A_(low) (900) to the low quality matrix A_(high,1)(902).

For the matrices A_(int er,1) (901) and A_(high,1) (902) shown in FIGS.9 b and 9 c, the encoder later exploits redundancy between the channelsfor which the encoder artificially increased inter-channel correlation,and the encoder need not instruct the decoder to perform anymulti-channel post-processing for those channels.

When the decoder has the ability to perform multi-channelpost-processing, the encoder can delegate reconstruction of the centerchannel to the decoder. If so, when the NER value n indicates poorquality coding, the encoder uses the matrix A_(high,2) (904) shown in 9e, with which the input center channel leaks into left and rightchannels. In the output, the center channel is zero, reducing the codingcomplexity.

$\begin{bmatrix}\left( {\frac{a}{1.5} + \frac{{.5} \cdot c}{1.5}} \right) \\\left( {\frac{b}{1.5} + \frac{{.5} \cdot c}{1.5}} \right) \\0 \\d \\\frac{e + f}{2} \\\frac{e + f}{2}\end{bmatrix} = {A_{{high},2} \cdot \begin{bmatrix}a \\b \\c \\d \\e \\f\end{bmatrix}}$When the encoder uses the pre-processing transform matrix A_(high,2)(904), the encoder (through the bitstream) instructs the decoder tocreate a phantom center by averaging the decoded left and rightchannels. Later multi-channel transformations in the encoder may exploitredundancy between the averaged back left and back right channels(without post-processing), or the encoder may instruct the decoder toperform some multi-channel post-processing for the back left and rightchannels. When the NER value n indicates intermediate quality coding,the encoder may use the intermediate matrix A_(int er,2) (903) shown inFIG. 9 d to transition between the matrices shown in FIGS. 9 a and 9 e.

FIG. 10 shows a technique (1000) for multi-channel pre-processing inwhich the transform matrix potentially changes on a frame-by-framebasis. Changing the transform matrix can lead to audible noise (e.g.,pops) in the final output if not handled carefully. To avoid introducingthe popping noise, the encoder gradually transitions from one transformmatrix to another between frames.

The encoder first sets (1010) the pre-processing transform matrix, asdescribed above. The encoder then determines (1020) if the matrix forthe current frame is the different than the matrix for the previousframe (if there was a previous frame). If the current matrix is the sameor there is no previous matrix, the encoder applies (1030) the matrix tothe input audio samples for the current frame. Otherwise, the encoderapplies (1040) a blended transform matrix to the input audio samples forthe current frame. The blending function depends on implementation. Inone implementation, at sample i in the current frame, the encoder uses ashort-term blended matrix A_(pre,i).

$\begin{matrix}{{A_{{pre},i} = {{\frac{{NumSamples} - i}{NumSamples}A_{{pre},{prev}}} + {\frac{i}{NumSamples}A_{{pre},{current}}}}},} & (6)\end{matrix}$where A_(pre,prev) and A_(pre,current) are the pre-processing matricesfor the previous and current frames, respectively, and NumSamples is thenumber of samples in the current frame. Alternatively, the encoder usesanother blending function to smooth discontinuities in thepre-processing transform matrices.

Then, the encoder encodes (1050) the multi-channel audio data for theframe, using techniques shown in FIG. 6 or other compression techniques.The encoder repeats the technique (1000) on a frame-by-frame basis.Alternatively, the encoder changes multi-channel pre-processing on someother basis.

IV. Tile Configuration

In some embodiments, an encoder such as the encoder (600) of FIG. 6groups windows of multi-channel audio into tiles for subsequentencoding. This gives the encoder flexibility to use different windowconfigurations for different channels in a frame, while also allowingmulti-channel transforms on various combinations of channels for theframe. A decoder such as the decoder (700) of FIG. 7 works with tilesduring decoding.

Each channel can have a window configuration independent of the otherchannels. Windows that have identical start and stop times areconsidered to be part of a tile. A tile can have one or more channels,and the encoder performs multi-channel transforms for channels in atile.

FIG. 11 a shows an example tile configuration (1100) for a frame ofstereo audio. In FIG. 11 a, each tile includes a single window. Nowindow in either channel of the stereo audio both starts and stops atthe same time as a window in the other channel.

FIG. 11 b shows an example tile configuration (1101) for a frame of 5.1channel audio. The tile configuration (1101) includes seven tiles,numbered 0 through 6. Tile 0 includes samples from channels 0, 2, 3, and4 and spans the first quarter of the frame. Tile 1 includes samples fromchannel 1 and spans the first half of the frame. Tile 2 includes samplesfrom channel 5 and spans the entire frame. Tile 3 is like tile 0, butspans the second quarter of the frame. Tiles 4 and 6 include samples inchannels 0, 2, and 3, and span the third and fourth quarters,respectively, of the frame. Finally, tile 5 includes samples fromchannels 1 and 4 and spans the last half of the frame. As shown in FIG.11 b, a particular tile can include windows in non-contiguous channels.

FIG. 12 shows a generalized technique (1200) for configuring tiles of aframe of multi-channel audio. The encoder sets (1210) the windowconfigurations for the channels in the frame, partitioning each channelinto variable-size windows to trade-off time resolution and frequencyresolution. For example, a partitioner/tile configurer of the encoderpartitions each channel independently of the other channels in theframe.

The encoder then groups (1220) windows from the different channels intotiles for the frame. For example, the encoder puts windows fromdifferent channels into a single tile if the windows have identicalstart positions and identical end positions. Alternatively, the encoderuses criteria other than or in addition to start/end positions todetermine which sections of different channels to group together into atile.

In one implementation, the encoder performs the tile grouping (1220)after (and independently from) the setting (1210) of the windowconfigurations for a frame. In other implementations, the encoderconcurrently sets (1210) window configurations and groups (1220) windowsinto tiles, for example, to favor time correlation (using longerwindows) or channel correlation (putting more channels into singletiles), or to control the number of tiles by coercing windows to fitinto a particular set of tiles.

The encoder then sends (1230) tile configuration information for theframe for output with the encoded audio data. For example, thepartitioner/tile configurer of the encoder sends tile size and channelmember information for the tiles to a MUX. Alternatively, the encodersends other information specifying the tile configurations. In oneimplementation, the encoder sends (1230) the tile configurationinformation after the tile grouping (1220). In other implementations,the encoder performs these actions concurrently.

FIG. 13 shows a technique (1300) for configuring tiles and sending tileconfiguration information for a frame of multi-channel audio accordingto a particular bitstream syntax. FIG. 13 shows the technique (1300)performed by the encoder to put information into the bitstream; thedecoder performs a corresponding technique (reading flags, gettingconfiguration information for particular tiles, etc.) to retrieve tileconfiguration information for the frame according to the bitstreamsyntax. Alternatively, the decoder and encoder use another syntax forone or more of the options shown in FIG. 13, for example, one that usesdifferent flags or different ordering.

The encoder initially checks (1310) if none of the channels in the frameare split into windows. If so, the encoder sends (1312) a flag bit(indicating that no channels are split), then exits. Thus, a single bitindicates if a given frame is one single tile or has multiple tiles.

On the other hand, if at least one channel is split into windows, theencoder checks (1320) whether all channels of the frame have the samewindow configuration. If so, the encoder sends (1322) a flag bit(indicating that all channels have the same window configuration—eachtile in the frame has all channels) and a sequence of tile sizes, thenexits. Thus, the single bit indicates if the channels all have the sameconfiguration (as in a conventional encoder bitstream) or have aflexible tile configuration.

If at least some channels have different window configurations, theencoder scans through the sample positions of the frame to identifywindows that have both the same start position and the same endposition. But first, the encoder marks (1330) all sample positions inthe frame as ungrouped. The encoder then scans (1340) for the nextungrouped sample position in the frame according to a channel/time scanpattern. In one implementation, the encoder scans through all channelsat a particular time looking for ungrouped sample positions, thenrepeats for the next sample position in time, etc. In otherimplementations, the encoder uses another scan pattern.

For the detected ungrouped sample position, the encoder groups (1350)like windows together in a tile. In particular, the encoder groupswindows that start at the start position of the window including thedetected ungrouped sample position, and that also end at the sameposition as the window including the detected ungrouped sample position.In the frame shown in FIG. 11 b, for example, the encoder would firstdetect the sample position at the beginning of channel 0. The encoderwould group the quarter-frame length windows from channels 0, 2, 3, and4 together in a tile since these windows each have the same startposition and same end position as the other windows in the tile.

The encoder then sends (1360) tile configuration information specifyingthe tile for output with the encoded audio data. The tile configurationinformation includes the tile size and a map indicating which channelswith ungrouped sample positions in the frame at that point are in thetile. The channel map includes one bit per channel possible for thetile. Based on the sequence of tile information, the decoder determineswhere a tile starts and ends in a frame. The encoder reduces bitrate forthe channel map by taking into account which channels can be present inthe tile. For example, the information for tile 0 in FIG. 11 b includesthe tile size and a binary pattern “101110” to indicate that channels 0,2, 3, and 4 are part of the tile. After that point, only samplepositions in channels 1 and 5 are ungrouped. So, the information fortile 1 includes the tile size and the binary pattern “10” to indicatethat channel 1 is part of the tile but channel 5 is not. This saves fourbits in the binary pattern. The tile information for tile 2 thenincludes only the tile size (and not the channel map), since channel 5is the only channel that can have a window starting in tile 2. The tileinformation for tile 3 includes the tile size and the binary pattern“1111” since the channels 1 and 5 have grouped positions in the rangefor tile 3. Alternatively, the encoder and decoder use another techniqueto signal channel patterns in the syntax.

The encoder then marks (1370) the sample positions for the windows inthe tile as grouped and determines (1380) whether to continue or not. Ifthere are no more ungrouped sample positions in the frame, the encoderexits. Otherwise, the encoder scans (1340) for the next ungrouped sampleposition in the frame according to the channel/time scan pattern.

V. Flexible Multi-Channel Transforms

In some embodiments, an encoder such as the encoder (600) of FIG. 6performs flexible multi-channel transforms that effectively takeadvantage of inter-channel correlation. A decoder such as the decoder(700) of FIG. 7 performs corresponding inverse multi-channel transforms.

Specifically, the encoder and decoder do one or more of the following toimprove multi-channel transformations in different situations.

1. The encoder performs the multi-channel transform after perceptualweighting, and the decoder performs the corresponding inversemulti-channel transform before inverse weighting. This reduces unmaskingof quantization noise across channels after the inverse multi-channeltransform.

2. The encoder and decoder group channels for multi-channel transformsto limit which channels get transformed together.

3. The encoder and decoder selectively turn multi-channel transformson/off at the frequency band level to control which bands aretransformed together.

4. The encoder and decoder use hierarchical multi-channel transforms tolimit computational complexity (especially in the decoder).

5. The encoder and decoder use pre-defined multi-channel transformmatrices to reduce the bitrate used to specify the transform matrices.

6. The encoder and decoder use quantized Givens rotation-basedfactorization parameters to specify multi-channel transform matrices forbit efficiency.

A. Multi-Channel Transform on Weighted Multi-Channel Audio

In some embodiments, the encoder positions the multi-channel transformafter perceptual weighting (and the decoder positions the inversemulti-channel transform before the inverse weighting) such that thecross-channel leaked signal is controlled, measurable, and has aspectrum like the original signal.

FIG. 14 shows a technique (1400) for performing one or moremulti-channel transforms after perceptual weighting in the encoder. Theencoder perceptually weights (1410) multi-channel audio, for example,applying weighting factors to multi-channel audio in the frequencydomain. In some implementations, the encoder applies both weightingfactors and per-channel quantization step modifiers to the multi-channelaudio data before the multi-channel transform(s).

The encoder then performs (1420) one or more multi-channel transforms onthe weighted audio data, for example, as described below. Finally, theencoder quantizes (1430) the multi-channel transformed audio data.

FIG. 15 shows a technique (1500) for performing an inverse-multi-channeltransform before inverse weighting in the decoder. The decoder performs(1510) one or more inverse multi-channel transforms on quantized audiodata, for example, as described below. In particular, the decodercollects samples from multiple channels at a particular frequency indexinto a vector x_(mc) and performs the inverse multi-channel transformA_(mc) to generate the output y_(mc).y _(mc) =A _(mc) ·x _(mc)  (7).

Subsequently, the decoder inverse quantizes and inverse weights (1520)the multi-channel audio, coloring the output of the inversemulti-channel transform with mask(s). Thus, leakage that occurs acrosschannels (due to quantization) is spectrally shaped so that the leakedsignal's audibility is measurable and controllable, and the leakage ofother channels in a given reconstructed channel is spectrally shapedlike the original uncorrupted signal of the given channel. (In someimplementations, per-channel quantization step modifiers also allow theencoder to make reconstructed signal quality approximately the sameacross all reconstructed channels.)

B. Channel Groups

In some embodiments, the encoder and decoder group channels formulti-channel transforms to limit which channels get transformedtogether. For example, in embodiments that use tile configuration, theencoder determines which channels within a tile correlate and groups thecorrelated channels. Alternatively, an encoder and decoder do not usetile configuration, but still group channels for frames or at some otherlevel.

FIG. 16 shows a technique (1600) for grouping channels of a tile formulti-channel transformation in one implementation. In the technique(1600), the encoder considers pair-wise correlations between the signalsof channels as well as correlations between bands in some cases.Alternatively, an encoder considers other and/or additional factors whengrouping channels for multi-channel transformation.

First, the encoder gets (1610) the channels for a tile. For example, inthe tile configuration shown in FIG. 11 b, tile 3 has four channels init: 0, 2, 3, and 4.

The encoder computes (1620) pair-wise correlations between the signalsin channels, and then groups (1630) channels accordingly. Suppose thatfor tile 3 of FIG. 11 b, channels 0 and 2 are pair-wise correlated, butneither of those channels is pair-wise correlated with channel 3 orchannel 4, and channel 3 is not pair-wise correlated with channel 4. Theencoder groups (1630) channels 0 and 2 together, puts channel 3 in aseparate group, and puts channel 4 in still another group.

A channel that is not pair-wise correlated with any of the channels in agroup may still be compatible with that group. So, for the channels thatare incompatible with a group, the encoder optionally checks (1640)compatibility at band level and adjusts (1650) the one or more groups ofchannels accordingly. In particular, this identifies channels that arecompatible with a group in some bands, but incompatible in some otherbands. For example, suppose that channel 4 of tile 3 in FIG. 11 b isactually compatible with channels 0 and 2 at most bands, but thatincompatibility in a few bands skews the pair-wise correlation results.The encoder adjusts (1650) the groups to put channels 0, 2, and 4together, leaving channel 3 in its own group. The encoder may alsoperform such testing when some channels are “overall” correlated, buthave incompatible bands. Turning off the transform at those incompatiblebands improves the correlation among the bands that actually getmulti-channel transform coded, and hence improves coding efficiency.

A channel in a given tile belongs to one channel group. The channels ina channel group need not be contiguous. A single tile may includemultiple channel groups, and each channel group may have a differentassociated multi-channel transform. After deciding which channels arecompatible, the encoder puts channel group information into thebitstream.

FIG. 17 shows a technique (1700) for retrieving channel groupinformation and multi-channel transform information for a tile from abitstream according to a particular bitstream syntax, irrespective ofhow the encoder computes channel groups. FIG. 17 shows the technique(1700) performed by the decoder to retrieve information from thebitstream; the encoder performs a corresponding technique to formatchannel group information and multi-channel transform information forthe tile according to the bitstream syntax. Alternatively, the decoderand encoder use another syntax for one or more of the options shown inFIG. 17.

First, the decoder initializes several variables used in the technique(1700). The decoder sets (1710) #ChannelsToVisit equal to the number ofchannels in the tile #ChannelsInTile and sets (1712) the number ofchannel groups #ChannelGroups to 0.

The decoder checks (1720) whether # ChannelsToVisit is greater than 2.If not, the decoder checks (1730) whether #ChannelsToVisit equals 2. Ifso, the decoder decodes (1740) the multi-channel transform for the groupof two channels, for example, using a technique described below. Thesyntax allows each channel group to have a different multi-channeltransform. On the other hand, if #ChannelsToVisit equal 1 or 0, thedecoder exits without decoding a multi-channel transform.

If #ChannelsToVisit is greater than 2, the decoder decodes (1750) thechannel mask for a group in the tile. Specifically, the decoder reads#ChannelsToVisit bits from the bitstream for the channel mask. Each bitin the channel mask indicates whether a particular channel is or is notin the channel group. For example, if the channel mask is “10110” thenthe tile includes 5 channels, and channels 0, 2, and 3 are in thechannel group.

The decoder then counts (1760) the number of channels in the group anddecodes (1770) the multi-channel transform for the group, for example,using a technique described below. The decoder updates (1780)#ChannelsToVisit by subtracting the counted number of channels in thecurrent channel group, increments (1790) #ChannelGroups, and checks(1720) whether the number of channels left to visit #ChannelsToVisit isgreater than 2.

Alternatively, in embodiments that do not use tile configurations, thedecoder retrieves channel group information and multi-channel transforminformation for a frame or at some other level.

C. Band On/Off Control for Multi-Channel Transform

In some embodiments, the encoder and decoder selectively turnmulti-channel transforms on/off at the frequency band level to controlwhich bands are transformed together. In this way, the encoder anddecoder selectively exclude bands that are not compatible inmulti-channel transforms. When the multi-channel transform is turned offfor a particular band, the encoder and decoder uses the identitytransform for that band, passing through the data at that band withoutaltering it.

The frequency bands are critical bands or quantization bands. The numberof frequency bands relates to the sampling frequency of the audio dataand the tile size. In general, the higher the sampling frequency orlarger the tile size, the greater the number of frequency bands.

In some implementations, the encoder selectively turns multi-channeltransforms on/off at the frequency band level for channels of a channelgroup of a tile. The encoder can turn bands on/off as the encoder groupschannels for a tile or after the channel grouping for the tile.Alternatively, an encoder and decoder do not use tile configuration, butstill turn multi-channel transforms on/off at frequency bands for aframe or at some other level.

FIG. 18 shows a technique (1800) for selectively including frequencybands of channels of a channel group in a multi-channel transform in oneimplementation. In the technique (1800), the encoder considers pair-wisecorrelations between the signals of the channels at a band to determinewhether to enable or disable the multi-channel transform for the band.Alternatively, an encoder considers other and/or additional factors whenselectively turning frequency bands on or off for a multi-channeltransform.

First, the encoder gets (1810) the channels for a channel group, forexample, as described with reference to FIG. 16. The encoder thencomputes (1820) pair-wise correlations between the signals in thechannels for different frequency bands. For example, if the channelgroup includes two channels, the encoder computes a pair-wisecorrelation at each frequency band. Or, if the channel group includesmore than two channels, the encoder computes pair-wise correlationsbetween some or all of the respective channel pairs at each frequencyband.

The encoder then turns (1830) bands on or off for the multi-channeltransform for the channel group. For example, if the channel groupincludes two channels, the encoder enables the multi-channel transformfor a band if the pair-wise correlation at the band satisfies aparticular threshold. Or, if the channel group includes more than twochannels, the encoder enables the multi-channel transform for a band ifeach or a majority of the pair-wise correlations at the band satisfies aparticular threshold. In alternative embodiments, instead of turning aparticular frequency band on or off for all channels, the encoder turnsthe band on for some channels and off for other channels.

After deciding which bands are included in multi-channel transforms, theencoder puts band on/off information into the bitstream.

FIG. 19 shows a technique (1900) for retrieving band on/off informationfor a multi-channel transform for a channel group of a tile from abitstream according to a particular bitstream syntax, irrespective ofhow the encoder decides whether to turn bands on or off. FIG. 19 showsthe technique (1900) performed by the decoder to retrieve informationfrom the bitstream; the encoder performs a corresponding technique toformat band on/off information for the channel group according to thebitstream syntax. Alternatively, the decoder and encoder use anothersyntax for one or more of the options shown in FIG. 19.

In some implementations, the decoder performs the technique (1900) aspart of the decoding of the multi-channel transform (1740 or 1770) ofthe technique (1700). Alternatively, the decoder performs the technique(1900) separately.

The decoder gets (1910) a bit and checks (1920) the bit to determinewhether all bands are enabled for the channel group. If so, the decoderenables (1930) the multi-channel transform for all bands of the channelgroup.

On the other hand, if the bit indicates all bands are not enabled forthe channel group, the decoder decodes (1940) the band mask for thechannel group. Specifically, the decoder reads a number of bits frombitstream, where the number is the number of bands for the channelgroup. Each bit in the band mask indicates whether a particular band ison or off for the channel group. For example, if the band mask is“111111110110000” then the channel group includes 15 bands, and bands 0,1, 2, 3, 4, 5, 6, 7, 9, and 10 are turned on for the multi-channeltransform. The decoder then enables (1950) the multi-channel transformfor the indicated bands.

Alternatively, in embodiments that do not use tile configurations, thedecoder retrieves band on/off information for a frame or at some otherlevel.

D. Hierarchical Multi-Channel Transforms

In some embodiments, the encoder and decoder use hierarchicalmulti-channel transforms to limit computational complexity, especiallyin the decoder. With the hierarchical transform, an encoder splits anoverall transformation into multiple stages, reducing the computationalcomplexity of individual stages and in some cases reducing the amount ofinformation needed to specify the multi-channel transform(s). Using thiscascaded structure, the encoder emulates the larger overall transformwith smaller transforms, up to some accuracy. The decoder performs acorresponding hierarchical inverse transform.

In some implementations, each stage of the hierarchical transform isidentical in structure and, in the bitstream, each stage is describedindependent of the one or more other stages. In particular, each stagehas its own channel groups and one multi-channel transform matrix perchannel group. In alternative implementations, different stages havedifferent structures, the encoder and decoder use a different bitstreamsyntax, and/or the stages use another configuration for channels andtransforms.

FIG. 20 shows a generalized technique (2000) for emulating amulti-channel transform using a hierarchy of simpler multi-channeltransforms. FIG. 20 shows an n stage hierarchy, where n is the number ofmulti-channel transform stages. For example, in one implementation, n is2. Alternatively, n is more than 2.

The encoder determines (2010) a hierarchy of multi-channel transformsfor an overall transform. The encoder decides the transform sizes (i.e.,channel group size) based on the complexity of the decoder that willperform the inverse transforms. Or the encoder considers target decoderprofile/decoder level or some other criteria.

FIG. 21 is a chart showing an example hierarchy (2100) of multi-channeltransforms. The hierarchy (2100) includes 2 stages. The first stageincludes N+1 channel groups and transforms, numbered from 0 to N; thesecond stage includes M+1 channel groups and transforms, numbered from 0to M. Each channel group includes 1 or more channels. For each of theN+1 transforms of the first stage, the input channels are somecombination of the channels input to the multi-channel transformer. Notall input channels must be transformed in the first stage. One or moreinput channels may pass through the first stage unaltered (e.g., theencoder may include such channels in an channel group that uses anidentity matrix.) For each of the M+1 transforms of the second stage,the input channels are some combination of the output channels from thefirst stage, including channels that may have passed through the firststage unaltered.

Returning to FIG. 20, the encoder performs (2020) the first stage ofmulti-channel transforms, performs the next stage of multi-channeltransforms, finally performing (2030) the n^(th) stage of multi-channeltransforms. A decoder performs corresponding inverse multi-channeltransforms during decoding.

In some implementations, the channel groups are the same at multiplestages of the hierarchy, but the multi-channel transforms are different.In such cases, and in certain other cases as well, the encoder maycombine frequency band on/off information for the multiple multi-channeltransforms. For example, suppose there are two multi-channel transformsand the same three channels in the channel group for each. The encodermay specify no transform/identity transform at both stages for band 0,only multi-channel transform stage 1 for band 1 (no stage 2 transform),only multi-channel transform stage 2 for band 2 (no stage 1 transform),both stages of multi-channel transforms for band 3, no transform at bothstages for band 4, etc.

FIG. 22 shows a technique (2200) for retrieving information for ahierarchy of multi-channel transforms for channel groups from abitstream according to a particular bitstream syntax. FIG. 22 shows thetechnique (2200) performed by the decoder to parse the bitstream; theencoder performs a corresponding technique to format the hierarchy ofmulti-channel transforms according to the bitstream syntax.Alternatively, the decoder and encoder use another syntax, for example,one that includes additional flags and signaling bits for more than twostages.

The decoder first sets (2210) a temporary value iTmp equal to the nextbit in the bitstream. The decoder then checks (2220) the value of thetemporary value, which signals whether or not the decoder should decode(2230) channel group and multi-channel transform information for a stage1 group.

After the decoder decodes (2230) channel group and multi-channeltransform information for a stage 1 group, the decoder sets (2240) iTmpequal to the next bit in the bitstream. The decoder again checks (2220)the value of iTmp, which signals whether or not the bitstream includeschannel group and multi-channel transform information for any more stage1 groups. Only the channel groups with non-identity transforms arespecified in the stage 1 portion of the bitstream; channels that are notdescribed in the stage 1 part of the bitstream are assumed to be part ofa channel group that uses an identity transform.

If the bistream includes no more channel group and multi-channeltransform information for stage 1 groups, the decoder decodes (2250)channel group and multi-channel transform information for all stage 2groups.

E. Pre-Defined or Custom Multi-Channel Transforms

In some embodiments, the encoder and decoder use pre-definedmulti-channel transform matrices to reduce the bitrate used to specifytransform matrices. The encoder selects from among multiple availablepre-defined matrix types and signals the selected matrix in thebitstream with a small number (e.g., 1, 2) of bits. Some types ofmatrices require no additional signaling in the bitstream, but othertypes of matrices require additional specification. The decoderretrieves the information indicating the matrix type and (if necessary)the additional information specifying the matrix.

In some implementations, the encoder and decoder use the followingpre-defined matrix types: identity, Hadamard, DCT type II, or arbitraryunitary. Alternatively, the encoder and decoder use different and/oradditional pre-defined matrix types.

FIG. 9 a shows an example of an identity matrix for 6 channels inanother context. The encoder efficiently specifies an identity matrix inthe bitstream using flag bits, assuming the number of dimensions for theidentity matrix are known to both the encoder and decoder from otherinformation (e.g., the number of channels in a group).

A Hadamard matrix has the following form.

$\begin{matrix}{{A_{Hadamard} = {\rho\begin{bmatrix}0.5 & {- 0.5} \\0.5 & 0.5\end{bmatrix}}},} & (8)\end{matrix}$where ρ is a normalizing scalar (√{square root over (2)}). The encoderefficiently specifies a Hadamard matrix for stereo data in the bitstreamusing flag bits.

A DCT type II matrix has the following form.

$\begin{matrix}{{A_{{DCT},{II}} = \begin{bmatrix}a_{0,0} & a_{0,1} & \cdots & a_{0,{N - 1}} \\a_{1,0} & a_{1,1} & \cdots & a_{1,{N - 1}} \\\cdots & \cdots & \cdots & \cdots \\a_{{N - 1},0} & a_{{N - 1},1} & \cdots & a_{{N - 1},{N - 1}}\end{bmatrix}},} & (9)\end{matrix}$where

$\begin{matrix}{{a_{n,m} = {k_{m} \cdot {\cos\left( \frac{{m \cdot \left( {n + 0.5} \right)}\pi}{N} \right)}}},} & (10)\end{matrix}$and where

$\begin{matrix}{k_{m} = \left\{ {\begin{matrix}\sqrt{\frac{1}{N}} & {m = 0} \\\sqrt{\frac{2}{N}} & {m > 0}\end{matrix}.} \right.} & (11)\end{matrix}$

For additional information about DCT type II matrices, see Rao et al.,Discrete Cosine Transform, Academic Press (1990). The DCT type II matrixcan have any size (i.e., work for any size channel group). The encoderefficiently specifies a DCT type II matrix in the bitstream using flagbits, assuming the number of dimensions for the DCT type II matrix areknown to both the encoder and decoder from other information (e.g., thenumber of channels in a group).

A square matrix A_(square) is unitary if its transposition is itsinverse.A _(square) ·A _(square) ^(T) =A _(square) ^(T) ·A _(square) =I  (12),where I is the identity matrix. The encoder uses arbitrary unitarymatrices to specify KLT transforms for effective redundancy removal. Theencoder efficiently specifies an arbitrary unitary matrix in thebitstream using flag bits and a parameterization of the matrix. In someimplementations, the encoder parameterizes the matrix using quantizedGivens factorizing rotations, as described below. Alternatively, theencoder uses another parameterization.

FIG. 23 shows a technique (2300) for selecting a multi-channel transformtype from among plural available types. The encoder selects a transformtype on a channel group-by-channel group basis or at some other level.

The encoder selects (2310) a multi-channel transform type from amongmultiple available types. For example, the available types includeidentity, Hadamard, DCT type II, and arbitrary unitary. Alternatively,the types include different and/or additional matrix types. The encoderuses an identity, Hadamard, or DCT type II matrix (rather than anarbitrary unitary matrix) if possible or if needed in order to reducethe bits needed to specify the transform matrix. For example, theencoder uses an identity, Hadamard, or DCT type II matrix if redundancyremoval is comparable or close enough (by some criteria) to redundancyremoval with the arbitrary unitary matrix. Or, the encoder uses anidentity, Hadamard, or DCT type II matrix if the encoder must reducebitrate. In a general situation, however, the encoder uses an arbitraryunitary matrix for the best compression efficiency.

The encoder then applies (2320) a multi-channel transform of theselected type to the multi-channel audio data.

FIG. 24 shows a technique (2400) for retrieving a multi-channeltransform type from among plural available types and performing aninverse multi-channel transform. The decoder retrieves transform typeinformation on a channel group-by-channel group basis or at some otherlevel.

The decoder retrieves (2410) a multi-channel transform type from amongmultiple available types. For example, the available types includeidentity, Hadamard, DCT type II, and arbitrary unitary. Alternatively,the types include different and/or additional matrix types. Ifnecessary, the decoder retrieves additional information specifying thematrix.

After reconstructing the matrix, the decoder applies (2420) an inversemulti-channel transform of the selected type to the multi-channel audiodata.

FIG. 25 shows a technique (2500) for retrieving multi-channel transforminformation for a channel group from a bitstream according to aparticular bitstream syntax. FIG. 25 shows the technique (2500)performed by the decoder to parse the bitstream; the encoder performs acorresponding technique to format the multi-channel transforminformation according to the bitstream syntax. Alternatively, thedecoder and encoder use another syntax, for example, one that usesdifferent flag bits, different ordering, or different transform types.

Initially, the decoder checks (2510) whether the number of channels inthe group # ChannelsInGroup is greater than 1. If not, the channel groupis for mono audio, and the decoder uses (2512) an identity transform forthe group.

If #ChannelsInGroup is greater than 1, the decoder checks (2520) whether#ChannelsInGroup is greater than 2. If not, the channel group is forstereo audio, and the decoder sets (2522) a temporary value iTmp equalto the next bit in the bitstream. The decoder then checks (2524) thevalue of the temporary value, which signals whether the decoder shoulduse (2530) a Hadamard transform for the channel group. If not, thedecoder sets (2526) iTmp equal to the next bit in the bitstream andchecks (2528) the value of iTmp, which signals whether the decodershould use (2550) an identity transform for the channel group. If not,the decoder decodes (2570) a generic unitary transform for the channelgroup.

If #ChannelsInGroup is greater than 2, the channel group is for surroundsound audio, and the decoder sets (2540) a temporary value iTmp equal tothe next bit in the bitstream. The decoder checks (2542) the value ofthe temporary value, which signals whether the decoder should use (2550)an identity transform of size #ChannelsInGroup for the channel group. Ifnot, the decoder sets (2560) iTmp equal to the next bit in the bitstreamand checks (2562) the value of iTmp. The bit signals whether the decodershould decode (2570) a generic unitary transform for the channel groupor use (2580) a DCT type II transform of size #ChannelsInGroup for thechannel group.

When the decoder uses a Hadamard, DCT type II, or generic unitarytransform matrix for the channel group, the decoder decodes (2590)multi-channel transform band on/off information for the matrix, thenexits.

F. Givens Rotation Representation of Transform Matrices

In some embodiments, the encoder and decoder use quantized Givensrotation-based factorization parameters to specify an arbitrary unitarytransform matrix for bit efficiency.

In general, a unitary transform matrix can be represented using Givensfactorizing rotations. Using this factorization, a unitary transformmatrix can be represented as:

$\begin{matrix}{A_{unitary} = {\Theta_{0,{N - 2}}\mspace{11mu}\cdots\mspace{11mu}\Theta_{0,1}\Theta_{0,0}\Theta_{1,{N - 3}}\mspace{11mu}\cdots\mspace{11mu}\Theta_{1,1}\Theta_{1,0}\mspace{11mu}\cdots\mspace{11mu}{\Theta_{{N - 2},0}\begin{bmatrix}\alpha_{0} & 0 & \cdots & 0 \\0 & \alpha_{1} & \cdots & 0 \\\cdots & \cdots & \cdots & \cdots \\0 & 0 & \cdots & \alpha_{N - 1}\end{bmatrix}}}} & (13)\end{matrix}$where α₁ is +1 or −1 (sign of rotation), and each Θ is of the form ofthe rotation matrix (2600) shown in FIG. 26. The rotation matrix (2600)is almost like an identity matrix, but has four sine/cosine terms withvarying positions. FIGS. 27 a-27 c show example rotation matrices forGivens rotations for representing a multi-channel transform matrix Thetwo cosine terms are always on the diagonal, the two sine terms are insame row/column as the cosine terms. Each Θ has one rotation angle, andits value can have a range

${- \frac{\pi}{2}} \leq \omega_{k} < {\frac{\pi}{2}.}$The number of such rotation matrices Θ needed to completely describe anN×N unitary matrix A_(unitary) is:

$\begin{matrix}{\frac{N\left( {N - 1} \right)}{2}.} & (14)\end{matrix}$

For additional information about Givens factorizing rotations, seeVaidyanathan, Multirate Systems and Filter Banks, Chapter 14.6,“Factorization of Unitary Matrices,” Prentice Hall (1993), herebyincorporated by reference.

In some embodiments, the encoder quantizes the rotation angles for theGivens factorization to reduce bitrate. FIG. 28 shows a technique (2800)for representing a multi-channel transform matrix using quantized Givensfactorizing rotations. Alternatively, an encoder or processing tool usesquantized Givens factorizing rotations to represent a unitary matrix forsome purpose other than multi-channel transformation of audio channels.

The encoder first computes (2810) an arbitrary unitary matrix for amulti-channel transform. The encoder then computes (2820) the Givensfactorizing rotations for the unitary matrix.

To reduce bitrate, the encoder quantizes (2830) the rotation angles. Inone implementation, the encoder uniformly quantizes each rotation angleto one of 64 (2⁶=64) possible values. The rotation signs are indicatedwith one bit each, so the encoder uses the following number of bits torepresent the N×N unitary matrix.

$\begin{matrix}{{{6 \cdot \frac{N\left( {N - 1} \right)}{2}} + N} = {{3N^{2}} - {2{N.}}}} & (15)\end{matrix}$This level of quantization allows the encoder to represent the N×Nunitary matrix for multi-channel transform with a very good degree ofprecision. Alternatively, the encoder uses some other level and/or typeof quantization.

FIG. 29 shows a technique (2900) for retrieving information for ageneric unitary transform for a channel group from a bitstream accordingto a particular bitstream syntax. FIG. 29 shows the technique (2900)performed by the decoder to parse the bitstream; the encoder performs acorresponding technique to format the information for the genericunitary transform according to the bitstream syntax. Alternatively, thedecoder and encoder use another syntax, for example, one that usesdifferent ordering or resolution for rotation angles.

First, the decoder initializes several variables used in the rest of thedecoding. Specifically, the decoder sets (2910) the number of angles todecode #AnglesToDecode based upon the number of channels in the channelgroup #ChannelsInGroup as shown in Equation 14. The decoder also sets(2912) the number of signs to decode #SignsToDecode based upon#ChannelsInGroup. The decoder also resets (2914, 2916) an angles decodedcounter iAnglesDecoded and a signs decoded counter iSignsDecoded.

The decoder checks (2920) whether there are any angles to decode and, ifso, sets (2922) the value for the next rotation angle, reconstructingthe rotation angle from the 6 bit quantized value.RotationAngle[iAnglesDecoded]=π*(getBits(6)−32)/64  (16).

The decoder then increments (2924) the angles decoded counter and checks(2920) whether there are any additional angles to decode.

When there are no more angles to decode, the decoder checks (2940)whether there are any additional signs to decode and, if so, sets (2942)the value for the next sign, reconstructing the sign from the 1 bitvalue.RotationSign[iSignsDecoded]=(2*getBits(1))−1  (17).

The decoder then increments (2944) the signs decoded counter and checks(2940) whether there are any additional signs to decode. When there areno more signs to decode, the decoder exits.

VI. Quantization and Weighting

In some embodiments, an encoder such as the encoder (600) of FIG. 6performs quantization and weighting on audio data using varioustechniques described below. For multi-channel audio configured intotiles, the encoder computes and applies quantization matrices forchannels of tiles, per-channel quantization step modifiers, and overallquantization tile factors. This allows the encoder to shape noiseaccording to an auditory model, balance noise between channels, andcontrol overall distortion.

A corresponding decoder such as the decoder (700) of FIG. 7 performsinverse quantization and inverse weighting. For multi-channel audioconfigured into tiles, the decoder decodes and applies overallquantization tile factors, per-channel quantization step modifiers, andquantization matrices for channels of tiles. The inverse quantizationand inverse weighting are fused into a single step.

A. Overall Tile Quantization Factor

In some embodiments, to control the quality and/or bitrate for the audiodata of a tile, a quantizer in an encoder computes a quantization stepsize Q_(t) for the tile. The quantizer may work in conjunction with arate/quality controller to evaluate different quantization step sizesfor the tile before selecting a tile quantization step size thatsatisfies the bitrate and/or quality constraints. For example, thequantizer and controller operate as described in U.S. patent applicationSer. No. 10/017,694, entitled “Quality and Rate Control Strategy forDigital Audio,” filed Dec. 14, 2001, hereby incorporated by reference.

FIG. 30 shows a technique (3000) for retrieving an overall tilequantization factor from a bitstream according to a particular bitstreamsyntax. FIG. 30 shows the technique (3000) performed by the decoder toparse the bitstream; the encoder performs a corresponding technique toformat the tile quantization factor according to the bitstream syntax.Alternatively, the decoder and encoder use another syntax, for example,one that works with different ranges for the tile quantization factor,uses different logic to encode the tile factor, or encodes groups oftile factors.

First, the decoder initializes (3010) the quantization step size Q_(t)for the tile. In one implementation, the decoder sets Q_(t) to:Q _(t)=90·ValidBitsPerSample/16  (18),where ValidBitsPerSample is a number 16≦ValidBitsPerSample≦24 that isset for the decoder or the audio clip, or set at some other level.

Next, the decoder gets (3020) six bits indicating the first modificationof Q_(t) relative to the initialized value of Q_(t), and stores thevalue −32≦Tmp≦31 in the temporary variable Tmp. The function SignExtend() determines a signed value from an unsigned value. The decoder adds(3030) the value of Tmp to the initialized value of Q_(t), thendetermines (3040) the sign of the variable Tmp, which is stored in thevariable SignofDelta.

The decoder checks (3050) whether the value of Tmp equals −32 or 31. Ifnot, the decoder exits. If the value of Tmp equals −32 or 31, theencoder may have signaled that Q_(t) should be further modified. Thedirection (positive or negative) of the further modification(s) isindicated by SignofDelta, and the decoder gets (3060) the next five bitsto determine the magnitude 0≦Tmp≦31 of the next modification. Thedecoder changes (3070) the current value of Q_(t) in the direction ofSignofDelta by the value of Tmp, then checks (3080) whether the value ofTmp is 31. If not, the decoder exits. If the value of Tmp is 31, thedecoder gets (3060) the next five bits and continues from that point.

In embodiments that do not use tile configurations, the encoder computesan overall quantization step size for a frame or other portion of audiodata.

B. Per-Channel Quantization Step Modifiers

In some embodiments, an encoder computes a quantization step modifierfor each channel in a tile: Q_(c0), Q_(c,1), . . . ,Q_(c,#ChannelsInTile−1). The encoder usually computes thesechannel-specific quantization factors to balance reconstruction qualityacross all channels. Even in embodiments that do not use tileconfigurations, the encoder can still compute per-channel quantizationfactors for the channels in a frame or other unit of audio data. Incontrast, previous quantization techniques such as those used in theencoder (100) of FIG. 1 use a quantization matrix element per band of awindow in a channel, but have no overall modifier for the channel.

FIG. 31 shows a generalized technique (3100) for computing per-channelquantization step modifiers for multi-channel audio data. The encoderuses several criteria to compute the quantization step modifiers. First,the encoder seeks approximately equal quality across all the channels ofreconstructed audio data. Second, if speaker positions are known, theencoder favors speakers that are more important to perception in typicaluses for the speaker configuration. Third, if speaker types are known,the encoder favors the better speakers in the speaker configuration.Alternatively, the encoder considers criteria other than or in additionto these criteria.

The encoder starts by setting (3110) quantization step modifiers for thechannels. In one implementation, the encoder sets (3110) the modifiersbased upon the energy in the respective channels. For example, for achannel with relatively more energy (i.e., louder) than the otherchannels, the quantization step modifiers for the other channels aremade relatively higher. Alternatively, the encoder sets (3110) themodifiers based upon other or additional criteria in an “open loop”estimation process. Or, the encoder can set (3110) the modifiers toequal values initially (relying on “closed loop” evaluation of resultsto converge on the final values for the modifiers).

The encoder quantizes (3120) the multi-channel audio data using thequantization step modifiers as well as other quantization (includingweighting) factors, if such other factors have not already been applied.

After subsequent reconstruction, the encoder evaluates (3130) thequality of the channels of reconstructed audio using NER or some otherquality measure. The encoder checks (3140) whether the reconstructedaudio satisfies the quality criteria (and/or other criteria) and, if so,exits. If not, the encoder sets (3110) new values for the quantizationstep modifiers, adjusting the modifiers in view of the evaluatedresults. Alternatively, for one-pass, open loop setting of the stepmodifiers, the encoder skips the evaluation (3130) and checking (3140).

Per-channel quantization step modifiers tend to change from window/tileto window/tile. The encoder codes the quantization step modifiers asliterals or variable length codes, and then packs them into thebitstream with the audio data. Or, the encoder uses some other techniqueto process the quantization step modifiers.

FIG. 32 shows a technique (3200) for retrieving per-channel quantizationstep modifiers from a bitstream according to a particular bitstreamsyntax. FIG. 32 shows the technique (3200) performed by the decoder toparse the bitstream; the encoder performs a corresponding technique(setting flags, packing data for the quantization step modifiers, etc.)to format the quantization step modifiers according to the bitstreamsyntax. Alternatively, the decoder and encoder use another syntax, forexample, one that works with different flags or logic to encode thequantization step modifiers.

FIG. 32 shows retrieval of per-channel quantization step modifiers for atile. Alternatively, in embodiments that do not use tiles, the decoderretrieves per-channel step modifiers for frames or other units of audiodata.

To start, the decoder checks (3210) whether the number of channels inthe tile is greater than 1. If not, the audio data is mono. The decodersets (3212) the quantization step modifier for the mono channel to 0 andexits.

For multi-channel audio, the decoder initializes several variables. Thedecoder gets (3220) bits indicating the number of bits per quantizationstep modifier (#BitsPerQ) for the tile. In one implementation, thedecoder gets three bits. The decoder then sets (3222) a channel counteriChannelsDone to 0.

The decoder checks (3230) whether the channel counter is less than thenumber of channels in the tile. If not, all channel quantization stepmodifiers for the tile have been retrieved, and the decoder exits.

On the other hand, if the channel counter is less than the number ofchannels in the tile, the decoder gets (3232) a bit and checks (3240)the bit to determine whether the quantization step modifier for thecurrent channel is 0. If so, the decoder sets (3242) the quantizationstep modifier for the current channel to 0.

If the quantization step modifier for the current channel is not 0, thedecoder checks (3250) whether # BitsPerQ is greater than 0 to determinewhether the quantization step modifier for the current channel is 1. Ifso, the decoder sets (3252) the quantization step modifier for thecurrent channel to 1.

If #BitsPerQ is greater than 0, the decoder gets the next #BitsPerQ bitsin the bitstream, adds 1 (since value of 0 triggers an earlier exitcondition), and sets (3260) the quantization step modifier for thecurrent channel to the result.

After the decoder sets the quantization step modifier for the currentchannel, the decoder increments (3270) the channel counter and checks(3230) whether the channel counter is less than the number of channelsin the tile.

C. Quantization Matrix Encoding and Decoding

In some embodiments, an encoder computes a quantization matrix for eachchannel in a tile. The encoder improves upon previous quantizationtechniques such as those used in the encoder (100) of FIG. 1 in severalways. For lossy compression of quantization matrices, the encoder uses aflexible step size for quantization matrix elements, which allows theencoder to change the resolution of the elements of quantizationmatrices. Apart from this feature, the encoder takes advantage oftemporal correlation in quantization matrix values during compression ofquantization matrices.

As previously discussed, a quantization matrix serves as a step sizearray, one step value per bark frequency band (or otherwise partitionedquantization band) for each channel in a tile. The encoder usesquantization matrices to “color” the reconstructed audio signal to havespectral shape comparable to that of the original signal. The encoderusually determines quantization matrices based on psychoacoustics andcompresses the quantization matrices to reduce bitrate. The compressionof quantization matrices can be lossy.

The techniques described in this section are described with reference toquantization matrices for channels of tiles. For notation, letQ_(m,iChannel,iBand) represent the quantization matrix element forchannel iChannel for the band iBand. In embodiments that do not use tileconfigurations, the encoder can still use a flexible step size forquantization matrix elements and/or take advantage of temporalcorrelation in quantization matrix values during compression.

1. Flexible Quantization Step Size for Mask Information

FIG. 33 shows a generalized technique (3300) for adaptively setting aquantization step size for quantization matrix elements. This allows theencoder to quantize mask information coarsely or finely. In oneimplementation, the encoder sets the quantization step size forquantization matrix elements on a channel-by-channel basis for a tile(i.e., matrix-by-matrix basis when each channel of the tile has amatrix). Alternatively, the encoder sets the quantization step size formask elements on a tile by-tile or frame-by-frame basis, for an entireaudio sequence, or at some other level.

The encoder starts by setting (3310) a quantization step size for one ormore mask(s). (The number of affected masks depends on the level atwhich the encoder assigns the flexible quantization step size.) In oneimplementation, the encoder evaluates the quality of reconstructed audioover some period of time and, depending on the result, selects thequantization step size to be 1, 2, 3, or 4 dB for mask information. Thequality measure evaluated by the encoder is NER for one or morepreviously encoded frames. For example, if the overall quality is poor,the encoder may set (3310) a higher value for the quantization step sizefor mask information, since resolution in the quantization matrix is notan efficient use of bitrate. On the other hand, if the overall qualityis good, the encoder may set (3310) a lower value for the quantizationstep size for mask information, since better resolution in thequantization matrix may efficiently improve perceived quality.Alternatively, the encoder uses another quality measure, evaluation overa different period, and/or other criteria in an open loop estimate forthe quantization step size. The encoder can also use different oradditional quantization step sizes for the mask information. Or, theencoder can skip the open loop estimate, instead relying on closed loopevaluation of results to converge on the final value for the step size.

The encoder quantizes (3320) the one or more quantization matrices usingthe quantization step size for mask elements, and weights and quantizesthe multi-channel audio data.

After subsequent reconstruction, the encoder evaluates (3330) thequality of the reconstructed audio using NER or some other qualitymeasure. The encoder checks (3340) whether the quality of thereconstructed audio justifies the current setting for the quantizationstep size for mask information. If not, the encoder may set (3310) ahigher or lower value for the quantization step size for maskinformation. Otherwise, the encoder exits. Alternatively, for one-pass,open loop setting of the quantization step size for mask information,the encoder skips the evaluation (3330) and checking (3340).

After selection, the encoder indicates the quantization step size formask information at the appropriate level in the bitstream.

FIG. 34 shows a generalized technique (3400) for retrieving an adaptivequantization step size for quantization matrix elements. The decoder canthus change the quantization step size for mask elements on achannel-by-channel basis for a tile, on a tile by-tile or frame-by-framebasis, for an entire audio sequence, or at some other level.

The decoder starts by getting (3410) a quantization step size for one ormore mask(s). (The number of affected masks depends on the level atwhich the encoder assigned the flexible quantization step size.) In oneimplementation, the quantization step size is 1, 2, 3, or 4 dB for maskinformation. Alternatively, the encoder and decoder use different oradditional quantization step sizes for the mask information.

The decoder then inverse quantizes (3420) the one or more quantizationmatrices using the quantization step size for mask information, andreconstructs the multi-channel audio data.

2. Temporal Prediction of Quantization Matrices

FIG. 35 shows a generalized technique (3500) for compressingquantization matrices using temporal prediction. With the technique(3500), the encoder takes advantage of temporal correlation in maskvalues. This reduces the bitrate associated with the quantizationmatrices.

FIGS. 35 and 36 show temporal prediction for quantization matrices in achannel of a frame of audio data. Alternatively, an encoder compressesquantization matrices using temporal prediction between multiple frames,over some other sequence of audio, or for a different configuration ofquantization matrices.

With reference to FIG. 35, the encoder gets (3510) quantization matricesfor a frame. The quantization matrices in a channel tend to be the samefrom window to window, making them good candidates for predictivecoding.

The encoder then encodes (3520) the quantization matrices using temporalprediction. For example, the encoder uses the technique (3600) shown inFIG. 36. Alternatively, the encoder uses another technique with temporalprediction.

The encoder determines (3530) whether there are any more matrices tocompress and, if not, exits. Otherwise, the encoder gets the nextquantization matrices. For example, the encoder checks whether matricesof the next frame are available for encoding.

FIG. 36 shows a more detailed technique (3600) for compressingquantization matrices in a channel using temporal prediction in oneimplementation. The temporal prediction uses a re-sampling processacross tiles of differing window sizes and uses run-level coding onprediction residuals to reduce bitrate.

The encoder starts (3610) the compression for next quantization matrixto be compressed and checks (3620) whether an anchor matrix isavailable, which usually depends on whether the matrix is the first inits channel. If an anchor matrix is not available, the encoder directlycompresses (3630) the quantization matrix. For example, the encoderdifferentially encodes the elements of the quantization matrix (wherethe difference for an element is relative to the element of the previousband) and assigns Huffman codes to the differentials. For the firstelement in the matrix (i.e., the mask element for the band 0), theencoder uses a prediction constant that depends on the quantization stepsize for the mask elements.PredConst=45/MaskQuantMultiplier_(iChannel)  (19).Alternatively, the encoder uses another compression technique for theanchor matrix.

The encoder then sets (3640) the quantization matrix as the anchormatrix for the channel of the frame. When the encoder uses tiles, thetile including the anchor matrix for a channel can be called the anchortile. The encoder notes the anchor matrix size or the tile size for theanchor tile, which may be used to form predictions for matrices with adifferent size.

On the other hand, if an anchor matrix is available, the encodercompresses the quantization matrix using temporal prediction. Theencoder computes (3650) a prediction for the quantization matrix basedupon the anchor matrix for the channel. If the quantization matrix beingcompressed has the same number of bands as the anchor matrix, theprediction is the elements of the anchor matrix. If the quantizationmatrix being compressed has a different number of bands than the anchormatrix, however, the encoder re-samples the anchor matrix to compute theprediction.

The re-sampling process uses the size of the quantization matrix beingcompressed/current tile size and the size of the anchor matrix/anchortile size.MaskPrediction[iBand]=AnchorMask[iScaledBand]  (20),where iScaledBand is the anchor matrix band that includes therepresentative (e.g., average) frequency of iBand. iBand is in terms ofthe current quantization matrix/current tile size, whereas iScaledBandis in terms of the anchor matrix/anchor tile size.

FIG. 37 illustrates one technique for re-sampling the anchor matrix whenthe encoder uses tiles. FIG. 37 shows an example mapping (3700) of bandsof a current tile to bands of an anchor tile to form a prediction.Frequencies in the middle of band boundaries (3720) of the quantizationmatrix in the current tile are mapped (3730) to frequencies of theanchor matrix in the anchor tile. The values for the mask prediction areset depending on where the mapped frequencies are relative to the bandboundaries (3710) of the anchor matrix in the anchor tile.Alternatively, the encoder uses temporal prediction relative to thepreceding quantization matrix in the channel or some other precedingmatrix, or uses another re-sampling technique.

Returning to FIG. 36, the encoder computes (3660) a residual for thequantization matrix relative to the prediction. Ideally, the predictionis perfect and the residual has no energy. If necessary, however, theencoder encodes (3670) the residual. For example, the encoder usesrun-level coding or another compression technique for the predictionresidual.

The encoder then determines (3680) whether there are any more matricesto be compressed and, if not, exits. Otherwise, the encoder gets (3610)the next quantization matrix and continues.

FIG. 38 shows a technique (3800) for retrieving and decodingquantization matrices compressed using temporal prediction according toa particular bitstream syntax. The quantization matrices are for thechannels of a single tile of a frame. FIG. 38 shows the technique (3800)performed by the decoder to parse information into the bitstream; theencoder performs a corresponding technique. Alternatively, the decoderand encoder use another syntax for one or more of the options shown inFIG. 38, for example, one that uses different flags or differentordering, or one that does not use tiles.

The decoder checks (3810) whether the encoder has reached the beginningof a frame. If so, the decoder marks (3812) all anchor matrices for theframe as being not set.

The decoder then checks (3820) whether the anchor matrix is available inthe channel of the next quantization matrix to be encoded. If no anchormatrix is available, the decoder gets (3830) the quantization step sizefor the quantization matrix for the channel. In one implementation, thedecoder gets the value 1, 2, 3, or 4 dB.MaskQuantMultiplier_(iChannel)=getBits(2)+1  (21).

The decoder then decodes (3832) the anchor matrix for the channel. Forexample, the decoder Huffman decodes differentially coded elements ofthe anchor matrix (where the difference for an element is relative tothe element of the previous band) and reconstructs the elements. For thefirst element, the decoder uses the prediction constant used in theencoder.PredConst=45/MaskQuantMultiplier_(iChannel)  (22).Alternatively, the decoder uses another decompression technique for theanchor matrix in a channel in the frame.

The decoder then sets (3834) the quantization matrix as the anchormatrix for the channel of the frame and sets the values of thequantization matrix for the channel to those of the anchor matrix.Q_(m,iChannel,iBand)=AnchorMask[iBand]  (23).

The decoder also notes the tile size for the anchor tile, which may beused to form predictions for matrices in tiles with a different sizethan the anchor tile.

On the other hand, if an anchor matrix is available for the channel, thedecoder decompresses the quantization matrix using temporal prediction.The decoder computes (3840) a prediction for the quantization matrixbased upon the anchor matrix for the channel. If the quantization matrixfor the current tile has the same number of bands as the anchor matrix,the prediction is the elements of the anchor matrix. If the quantizationmatrix for the current tile has a different number of bands as theanchor matrix, however, the encoder re-samples the anchor matrix to getthe prediction, for example, using the current tile size and anchor tilesize as shown in FIG. 37.MaskPrediction[iBand]=AnchorMask[iScaledBand]  (24).

Alternatively, the decoder uses temporal prediction relative to thepreceding quantization matrix in the channel or some other precedingmatrix, or uses another re-sampling technique.

The decoder gets (3842) the next bit in the bitstream and checks (3850)whether the bitstream includes a residual for the quantization matrix.If there is no mask update for this channel in the current tile, themask prediction residual is 0, so:Q_(m,iChannel,iBand)=MaskPrediction[iBand]  (25).

On the other hand, if there is a prediction residual, the decoderdecodes (3852) the residual, for example, using run-level decoding orsome other decompression technique. The decoder then adds (3854) theprediction residual to the prediction to reconstruct the quantizationmatrix. For example, the addition is a simple scalar addition on aband-by-band basis to get the element for band iBand for the currentchannel iChannel:Q_(m,iChannel,iBand)=MaskPrediction[iBand]+MaskPredResidual[iBand]  (26).

The decoder then checks (3860) whether quantization matrices for allchannels in the current tile have been decoded and, if so, exits.Otherwise, the decoder continues decoding for the next quantizationmatrix in the current tile.

D. Combined Inverse Quantization and Inverse Weighting

Once the decoder retrieves all the necessary quantization and weightinginformation, the decoder inverse quantizes and inverse weights the audiodata. In one implementation, the decoder performs the inversequantization and inverse weighting in one step, which is shown in twoequations below for the sake of clear printing.CombinedQ=Q _(t) +Q _(c,iChannel)−(Max(Q _(m,iChannel,*))−Q_(m,iChannel,iBand))·MaskQuantMultiplier_(iChannel)  (27a),y _(iqw) [n]=10 CombinedQ/20·x _(iqw) [n]  (27b).where x_(iqw) is the input (e.g., inverse MC-transformed coefficient) ofchannel iChannel, and n is a coefficient index in band iBand.Max(Q_(m,iChannel,*)) is the maximum mask value for the channel iChannelover all bands. (The difference between the largest and smallestweighting factors for a mask is typically much less than the range ofpotential values for mask elements, so the amount of quantizationadjustment per weighting factor is computed relative to the maximum.)MaskQuantMultiplier_(iChannel) is the mask quantization step multiplierfor the quantization matrix of channel iChannel, and y_(iqw) is theoutput of this step.

Alternatively, the decoder performs the inverse quantization andweighting separately or using different techniques.

VII. Multi-Channel Post-Processing

In some embodiments, a decoder such as the decoder (700) of FIG. 7performs multi-channel post-processing on reconstructed audio samples inthe time-domain.

The multi-channel post-processing can be used for many differentpurposes. For example, the number of decoded channels may be less thanthe number of channels for output (e.g., because the encoder dropped oneor more input channels or multi-channel transformed channels to reducecoding complexity or buffer fullness). If so, a multi-channelpost-processing transform can be used to create one or more phantomchannels based on actual data in the decoded channels. Or, even if thenumber of decoded channels equals the number of output channels, thepost-processing transform can be used for arbitrary spatial rotation ofthe presentation, remapping of output channels between speakerpositions, or other spatial or special effects. Or, if the number ofdecoded channels is greater than the number of output channels (e.g.,playing surround sound audio on stereo equipment), the post-processingtransform can be used to “fold-down” channels. In some embodiments, thefold-down coefficients potentially vary over time—the multi-channelpost-processing is bitstream-controlled. The transform matrices forthese scenarios and applications can be provided or signaled by theencoder.

FIG. 39 shows a generalized technique (3900) for multi-channelpost-processing. The decoder decodes (3910) encoded multi-channel audiodata (3905) using techniques shown in FIG. 7 or other decompressiontechniques, producing reconstructed time-domain multi-channel audio data(3915).

The decoder then performs (3920) multi-channel post-processing on thetime-domain multi-channel audio data (3915). For example, when theencoder produces M decoded channels and the decoder outputs N channels,the post-processing involves a general M to N transform. The decodertakes M co-located (in time) samples, one from each of the reconstructedM coded channels, then pads any channels that are missing (i.e., the N−Mchannels dropped by the encoder) with zeros. The decoder multiplies theN samples with a matrix A_(post).y _(post) =A _(post) ·x _(post)  (28),where x_(post) and y_(post) are the N channel input to and the outputfrom the multi-channel post-processing, A_(post) is a general N×Ntransform matrix, and x_(post) is padded with zeros to match the outputvector length N.

The matrix A_(post) can be a matrix with pre-determined elements, or itcan be a general matrix with elements specified by the encoder. Theencoder signals the decoder to use a pre-determined matrix (e.g., withone or more flag bits) or sends the elements of a general matrix to thedecoder, or the decoder may be configured to always use the same matrixA_(post). The matrix A_(post) need not possess special characteristicssuch as being as symmetric or invertible. For additional flexibility,the multi-channel post-processing can be turned on/off on aframe-by-frame or other basis (in which case, the decoder may use anidentity matrix to leave channels unaltered).

FIG. 40 shows an example matrix A_(P-center) (4000) used to create aphantom center channel from left and right channels in a 5.1 channelplayback environment with the channels ordered as shown in FIG. 4. Theexample matrix A_(P-center) (4000) passes the other channels throughunaltered. The decoder gets samples co-located in time from the left,right, sub-woofer, back left, and back right channels and pads thecenter channel with 0s. The decoder then multiplies the six inputsamples by the matrix A_(P-center) (4000).

$\begin{matrix}{\begin{bmatrix}a \\b \\\frac{a + b}{2} \\d \\e \\f\end{bmatrix} = {A_{P - {Center}} \cdot {\begin{bmatrix}a \\b \\0 \\d \\e \\f\end{bmatrix}.}}} & (29)\end{matrix}$

Alternatively, the decoder uses a matrix with different coefficients ora different number of channels. For example, the decoder uses a matrixto create phantom channels in a 7.1 channel, 9.1 channel, or some otherplayback environment from coded channels for 5.1 multi-channel audio.

FIG. 41 shows a technique (4100) for multi-channel post-processing inwhich the transform matrix potentially changes on a frame-by-framebasis. Changing the transform matrix can lead to audible noise (e.g.,pops) in the final output if not handled carefully. To avoid introducingthe popping noise, the decoder gradually transitions from one transformmatrix to another between frames.

The decoder first decodes (4110) the encoded multi-channel audio datafor a frame, using techniques shown in FIG. 7 or other decompressiontechniques, and producing reconstructed time-domain multi-channel audiodata. The decoder then gets (4120) the post-processing matrix for theframe, for example, as shown in FIG. 42.

The decoder determines (4130) if the matrix for the current frame is thedifferent than the matrix for the previous frame (if there was aprevious frame). If the current matrix is the same or there is noprevious matrix, the decoder applies (4140) the matrix to thereconstructed audio samples for the current frame. Otherwise, thedecoder applies (4150) a blended transform matrix to the reconstructedaudio samples for the current frame. The blending function depends onimplementation. In one implementation, at sample i in the current frame,the decoder uses a short-term blended matrix A_(post,i).

$\begin{matrix}{{A_{{post},i} = {{\frac{{NumSamples} - i}{NumSamples}A_{{post},{prev}}} + {\frac{i}{NumSamples}A_{{post},{current}}}}},} & (30)\end{matrix}$where A_(post,prev) and A_(post,current) are the post-processingmatrices for the previous and current frames, respectively, andNumSamples is the number of samples in the current frame. Alternatively,the decoder uses another blending function to smooth discontinuities inthe post-processing transform matrices.

The decoder repeats the technique (4100) on a frame-by-frame basis.Alternatively, the decoder changes multi-channel post-processing on someother basis.

FIG. 42 shows a technique (4200) for identifying and retrieving atransform matrix for multi-channel post-processing according to aparticular bitstream syntax. The syntax allows specification pre-definedtransform matrices as well as custom matrices for multi-channelpost-processing. FIG. 42 shows the technique (4200) performed by thedecoder to parse the bitstream; the encoder performs a correspondingtechnique (setting flags, packing data for elements, etc.) to format thetransform matrix according to the bitstream syntax. Alternatively, thedecoder and encoder use another syntax for one or more of the optionsshown in FIG. 42, for example, one that uses different flags ordifferent ordering.

First, the decoder determines (4210) if the number of channels #Channelsis greater than 1. If #Channels is 1, the audio data is mono, and thedecoder uses (4212) an identity matrix (i.e., performs no multi-channelpost-processing per se).

On the other hand, if #Channels is >1, the decoder sets (4220) atemporary value iTmp equal to the next bit in the bitstream. The decoderthen checks (4230) the value of the temporary value, which signalswhether or not the decoder should use (4232) an identity matrix.

If the decoder uses something other than an identity matrix for themulti-channel audio, the decoder sets (4240) the temporary value iTmpequal to the next bit in the bitstream. The decoder then checks (4250)the value of the temporary value, which signals whether or not thedecoder should use (4252) a pre-defined multi-channel transform matrix.If the decoder uses (4252) a pre-defined matrix, the decoder may get oneor more additional bits from the bitstream (not shown) that indicatewhich of several available pre-defined matrices the decoder should use.

If the decoder does not use a pre-defined matrix, the decoderinitializes various temporary values for decoding a custom matrix. Thedecoder sets (4260) a counter iCoefsDone for coefficients done to 0 andsets (4262) the number of coefficients #CoefsToDo to decode to equal thenumber of elements in the matrix (#Channels²). For matrices known tohave particular properties (e.g., symmetric), the number of coefficientsto decode can be decreased. The decoder then determines (4270) whetherall coefficients have been retrieved from the bitstream and, if so,ends. Otherwise, the decoder gets (4272) the value of the next elementA[iCoefsDone] in the matrix and increments (4274) iCoefsDone. The wayelements are coded and packed into the bitstream is implementationdependent. In FIG. 42, the syntax allows four bits of precision perelement of the transform matrix, and the absolute value of each elementis less than or equal to 1. In other implementations, the precision perelement is different, the encoder and decoder use compression to exploitpatterns of redundancy in the transform matrix, and/or the syntaxdiffers in some other way.

Having described and illustrated the principles of our invention withreference to described embodiments, it will be recognized that thedescribed embodiments can be modified in arrangement and detail withoutdeparting from such principles. It should be understood that theprograms, processes, or methods described herein are not related orlimited to any particular type of computing environment, unlessindicated otherwise. Various types of general purpose or specializedcomputing environments may be used with or perform operations inaccordance with the teachings described herein. Elements of thedescribed embodiments shown in software may be implemented in hardwareand vice versa.

In view of the many possible embodiments to which the principles of ourinvention may be applied, we claim as our invention all such embodimentsas may come within the scope and spirit of the following claims andequivalents thereto.

1. In an audio encoder, a computer-implemented method comprising:receiving multi-channel audio data; selecting a multi-channel transformfrom among plural available types of multi-channel transforms, whereineach of the plural available types of multi-channel transforms specifiesa different transformation across plural channels for co-located valuesin the plural channels; selectively turning the selected transformon/off at plural frequency bands; and performing the selected transformon the audio data at one or more of the plural frequency bands at whichthe selected transform is on, wherein the encoder performs no transformor an identity transform on the audio data at zero or more of the pluralfrequency bands at which the selected transform is off.
 2. The method ofclaim 1 wherein the multi-channel audio data is in two channels.
 3. Themethod of claim 1 wherein the multi-channel audio data is in more thantwo channels.
 4. The method of claim 1 further comprising outputting amask including one bit for each of the plural frequency bands.
 5. Themethod of claim 1 further comprising outputting a single bit and, if theselected transform is not turned on at all of the plural frequencybands, a mask including one bit for each of the plural frequency bands.6. The method of claim 1 wherein the encoder selectively turns theselected transform on/off based at least in part upon channelcorrelation measurements at the plural frequency bands.
 7. In an audiodecoder, a computer-implemented method comprising: receiving encodedmulti-channel audio data; selecting an inverse multi-channel transformfrom among plural available types of inverse multi-channel transforms,wherein each of the plural available types of inverse multi-channeltransforms specifies a different transformation across plural channelsfor co-located values in the plural channels; retrieving information forfrequency band on/off selections for plural frequency bands; andperforming the selected transform on the audio data at one or more ofthe plural frequency bands at which the selected transform is on,wherein the decoder performs no transform or an identity transform onthe audio data at zero or more of the plural frequency bands at whichthe selected transform is off.
 8. The method of claim 7 wherein themulti-channel audio data is in two channels.
 9. The method of claim 7wherein the multi-channel audio data is in more than two channels. 10.The method of claim 7 wherein the retrieved information comprises a maskincluding one bit for each of the plural frequency bands.
 11. The methodof claim 7 wherein the retrieved information comprises a single bit and,if the selected transform is not turned on at all of the pluralfrequency bands, a mask including one bit for each of the pluralfrequency bands.
 12. In an audio encoder, a computer-implemented methodcomprising: receiving multi-channel audio data; selecting amulti-channel transform from among plural available types ofmulti-channel transforms, wherein the plural available types includethree or more pre-defined transforms, and wherein each of the pluralavailable types of multi-channel transforms specifies a differenttransformation across plural channels for co-located values in theplural channels; and performing the selected transform on the audiodata.
 13. The method of claim 12 wherein the multi-channel audio data isin two channels.
 14. The method of claim 12 wherein the multi-channelaudio data is in more than two channels.
 15. The method of claim 12wherein the pre-defined transforms include a DCT variant and a Hadamardtransform.
 16. The method of claim 12 wherein the plural available typesfurther include a general unitary transform.
 17. The method of claim 12further comprising outputting information indicating the selectedtransform.
 18. In an audio encoder, a computer-implemented methodcomprising: receiving multi-channel audio data; selecting amulti-channel transform from among plural available types ofmulti-channel transforms, wherein the plural available types includeplural pre-defined transforms and at least one custom transform, andwherein each of the plural available types of multi-channel transformsspecifies a different transformation across plural channels forco-located values in the plural channels; and performing the selectedtransform on the audio data.
 19. The method of claim 18 wherein themulti-channel audio data is in two channels.
 20. The method of claim 18wherein the multi-channel audio data is in more than two channels. 21.The method of claim 18 further comprising outputting informationindicating the selected transform.
 22. The method of claim 21 whereinthe output information includes information for individual elements ofthe selected transform.
 23. The method of claim 18 wherein the encoderselects one of the plural pre-defined transforms if performance of theselected pre-defined transform is suitably close to performance of thecustom transform in terms of redundancy removal.
 24. In an audiodecoder, a computer-implemented method comprising: receiving encodedmulti-channel audio data; selecting an inverse multi-channel transformfrom among plural available types of inverse multi-channel transforms,wherein the plural available types include three or more pre-definedtransforms, and wherein each of the plural available types of inversemulti-channel transforms specifies a different transformation acrossplural channels for co-located values in the plural channels; andperforming the selected transform on the audio data.
 25. The method ofclaim 24 wherein the multi-channel audio data is in two channels. 26.The method of claim 24 wherein the multi-channel audio data is in morethan two channels.
 27. The method of claim 24 wherein the pre-definedtransforms include a DCT variant and a Hadamard transform.
 28. Themethod of claim 24 further comprising, before the selecting, retrievinginformation indicating the selected transform.
 29. The method of claim28 wherein the plural available types further include a customtransform, wherein the retrieved information includes one or moresignals to select the custom transform, and wherein the retrievedinformation further includes information for individual elements of thecustom transform.
 30. In an audio decoder, a computer-implemented methodcomprising: receiving encoded multi-channel audio data; selecting aninverse multi-channel transform from among plural available types ofinverse multi-channel transforms, wherein the plural available typesinclude plural pre-defined transforms and at least one custom transform,and wherein each of the plural available types of inverse multi-channeltransforms specifies a different transformation across plural channelsfor co-located values in the plural channels; and performing theselected transform on the audio data.
 31. The method of claim 30 whereinthe multi-channel audio data is in two channels.
 32. The method of claim30 wherein the multi-channel audio data is in more than two channels.33. The method of claim 30 further comprising, before the selecting,retrieving information indicating the selected transform.
 34. The methodof claim 33 wherein the retrieved information includes one or moresignals to select the custom transform, and wherein the retrievedinformation further includes information for individual elements of thecustom transform.
 35. A computer-readable medium storingcomputer-executable instructions for causing a computer programmedthereby to perform a method in an audio encoder, the method comprising:receiving multi-channel audio data; selecting a multi-channel transformfrom among plural available types of multi-channel transforms, whereineach of the plural available types of multi-channel transforms specifiesa different transformation across plural channels for co-located valuesin the plural channels; selectively turning the selected transformon/off at plural frequency bands; and performing the selected transformon the audio data at one or more of the plural frequency bands at whichthe selected transform is on, wherein the encoder performs no transformor an identity transform on the audio data at zero or more of the pluralfrequency bands at which the selected transform is off.
 36. Acomputer-readable medium storing computer-executable instructions forcausing a computer programmed thereby to perform a method in an audiodecoder, the method comprising: receiving encoded multi-channel audiodata; selecting an inverse multi-channel transform from among pluralavailable types of inverse multi-channel transforms, wherein each of theplural available types of inverse multi-channel transforms specifies adifferent transformation across plural channels for co-located values inthe plural channels; retrieving information for frequency band on/offselections for plural frequency bands; and performing the selectedtransform on the audio data at one or more of the plural frequency bandsat which the selected transform is on, wherein the decoder performs notransform or an identity transform on the audio data at zero or more ofthe plural frequency bands at which the selected transform is off.
 37. Acomputer-readable medium storing computer-executable instructions forcausing a computer programmed thereby to perform a method in an audioencoder, the method comprising: receiving multi-channel audio data;selecting a multi-channel transform from among plural available types ofmulti-channel transforms, wherein the plural available types includethree or more pre-defined transforms, and wherein each of the pluralavailable types of multi-channel transforms specifies a differenttransformation across plural channels for co-located values in theplural channels; and performing the selected transform on the audiodata.
 38. A computer-readable medium storing computer-executableinstructions for causing a computer programmed thereby to perform amethod in an audio encoder, the method comprising: receivingmulti-channel audio data; selecting a multi-channel transform from amongplural available types of multi-channel transforms, wherein the pluralavailable types include plural pre-defined transforms and at least onecustom transform, and wherein each of the plural available types ofmulti-channel transforms specifies a different transformation acrossplural channels for co-located values in the plural channels; andperforming the selected transform on the audio data.
 39. Acomputer-readable medium storing computer-executable instructions forcausing a computer programmed thereby to perform a method in an audiodecoder, the method comprising: receiving encoded multi-channel audiodata; selecting an inverse multi-channel transform from among pluralavailable types of inverse multi-channel transforms, wherein the pluralavailable types include three or more pre-defined transforms, andwherein each of the plural available types of inverse multi-channeltransforms specifies a different transformation across plural channelsfor co-located values in the plural channels; and performing theselected transform on the audio data.
 40. A computer-readable mediumstoring computer-executable instructions for causing a computerprogrammed thereby to perform a method in an audio decoder, the methodcomprising: receiving encoded multi-channel audio data; selecting aninverse multi-channel transform from among plural available types ofinverse multi-channel transforms, wherein the plural available typesinclude plural pre-defined transforms and at least one custom transform,and wherein each of the plural available types of inverse multi-channeltransforms specifies a different transformation across plural channelsfor co-located values in the plural channels; and performing theselected transform on the audio data.
 41. An audio encoder, comprising:means for receiving multi-channel audio data; means for selecting amulti-channel transform from among plural available types ofmulti-channel transforms, wherein each of the plural available types ofmulti-channel transforms specifies a different transformation acrossplural channels for co-located values in the plural channels; means forselectively turning the selected transform on/off at plural frequencybands; and means for performing the selected transform on the audio dataat one or more of the plural frequency bands at which the selectedtransform is on, wherein the encoder performs no transform or anidentity transform on the audio data at zero or more of the pluralfrequency bands at which the selected transform is off.
 42. An audiodecoder, comprising: means for receiving encoded multi-channel audiodata; means for selecting an inverse multi-channel transform from amongplural available types of inverse multi-channel transforms, wherein eachof the plural available types of inverse multi-channel transformsspecifies a different transformation across plural channels forco-located values in the plural channels; means for retrievinginformation for frequency band on/off selections for plural frequencybands; and means for performing the selected transform on the audio dataat one or more of the plural frequency bands at which the selectedtransform is on, wherein the decoder performs no transform or anidentity transform on the audio data at zero or more of the pluralfrequency bands at which the selected transform is off.
 43. An audioencoder, comprising: means for receiving multi-channel audio data; meansfor selecting a multi-channel transform from among plural availabletypes of multi-channel transforms, wherein the plural available typesinclude three or more pre-defined transforms, and wherein each of theplural available types of multi-channel transforms specifies a differenttransformation across plural channels for co-located values in theplural channels; and means for performing the selected transform on theaudio data.
 44. An audio encoder, comprising: means for receivingmulti-channel audio data; means for selecting a multi-channel transformfrom among plural available types of multi-channel transforms, whereinthe plural available types include plural pre-defined transforms and atleast one custom transform, and wherein each of the plural availabletypes of multi-channel transforms specifies a different transformationacross plural channels for co-located values in the plural channels; andmeans for performing the selected transform on the audio data.
 45. Anaudio decoder, comprising: means for receiving encoded multi-channelaudio data; means for selecting an inverse multi-channel transform fromamong plural available types of inverse multi-channel transforms,wherein the plural available types include three or more pre-definedtransforms, and wherein each of the plural available types of inversemulti-channel transforms specifies a different transformation acrossplural channels for co-located values in the plural channels; and meansfor performing the selected transform on the audio data.
 46. An audiodecoder, comprising: means for receiving encoded multi-channel audiodata; means for selecting an inverse multi-channel transform from amongplural available types of inverse multi-channel transforms, wherein theplural available types include plural pre-defined transforms and atleast one custom transform, and wherein each of the plural availabletypes of inverse multi-channel transforms specifies a differenttransformation across plural channels for co-located values in theplural channels; and means for performing the selected transform on theaudio data.
 47. The method of claim 1 wherein the co-located values inthe plural channels are frequency coefficients at a frequency index inthe plural channels.
 48. The method of claim 7 wherein the co-locatedvalues in the plural channels are frequency coefficients at a frequencyindex in the plural channels.
 49. The method of claim 12 wherein theco-located values in the plural channels are frequency coefficients at afrequency index in the plural channels.
 50. The method of claim 18wherein the co-located values in the plural channels are frequencycoefficients at a frequency index in the plural channels.